=] 
= 
on 
— — 
7; 
| mm 
Ao 
J 5 
i Ga 
N a 
= 


GABRIEL HALLEWY 


WHEN ROBOTS KILL 


ARTIFICIAL INTELLIGENCE UNDER CRIMINAL LAW 


Northeaster N UNiversity Press - Boston 


Northeastern University Press 
An imprint of University Press of New England 
www.upne.com 
© 2013 Northeastern University 
All rights reserved 
Manufactured in the United States of America 
Designed by Vicki Kuskowski 
Typeset in Melior by Copperline Book Services, Inc. 


University Press of New England is a member of the Green Press 
Initiative. The paper used in this book meets their minimum 
requirement for recycled paper. 


For permission to reproduce any of the material in this book, 
contact Permissions, University Press of New England, 
One Court Street, Suite 250, Lebanon NH 03766; 
or visit www.upne.com 


Library of Congress Cataloging-in-Publication Data 
Halevi, Gavri’el. 
When robots kill : artificial intelligence under criminal law / 
Gabriel Hallevy. 
pages cm. 
Includes bibliographical references and index. 
ISBN 978—-1—55553—801—9 (cloth) — 
ISBN 978—1—55553—805—7 (pbk.) — 
ISBN 978—1—55553—806—4 (ebook) (print) 

1. Criminal liability. 2. Artificial intelligence—Law and 
legislation—Criminal provisions. I. Title. 
K5070.H35 2013 
345'.0252—dc23 2012033518 


54321 


to my dearf amil y 


And God said, let us make man in our image, after our 
likeness: and let them have dominion over the fish of the sea, 
and over the fowl of the air, and over the cattle, 
and over all the earth, and over every creeping thing that 
creepeth upon the earth. So God created man in 
his own image, in the image of God created he him; 
male and female created he them. 

—GeNesis 1:26-27 
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In 1981, a thirty-seven-year-old japanese employee in a 
motorcycle factory was killed by an artificial intelligence 
robot working near him. The robot erroneously identified 
the employee as a threat to its mission, and calculated that 
the most efficient way to eliminate the threat was to push the 
worker into an adjacent machine. Usingits very powerful 
hydraulic arm, the robot smashed the surprised worker into 
the operating machine, killing him instantly, after which it 
resumed its duties without further interference. This is not 
science fiction, and the legal question is this: Who is to be 
held criminally liable for this homicide?" 


In most developed countries, unmanned vehicles, surgical robots, indus- 
trial robots, trading algorithms, personal robots, and other artificial intel- 
ligence (AT) entities are in common use. Such use may be personal, medi- 
cal, military, commercial, or industrial. The question of criminal liability 
arises when the unmanned vehicle is involved in car accidents, the surgi- 
cal robot is involved in surgical errors, the trading algorithm is involved 
in fraud, and so on. Who is to be held criminally liable for these offenses: 
the manufacturer, the programmer, the user, or the AI entity itself? 

The technological world is changing rapidly. Robots and computers 
are more frequently replacing humans in performing simple activities. 
As long as humanity used computers as mere tools, there was no signifi- 
cant difference between computers and screwdrivers, cars, or telephones. 
But as computers became increasingly sophisticated, we started saying 
that they “think” for us. Problems began when computers evolved from 
“thinking machines” (devices programmed to perform specific thought 
processes, such as computing) into thinking machines without the quota- 
tion marks— in other words, artificially intelligent. Artificial intelligence 
is the ability of a machine to imitate intelligent behavior. 

Artificial intelligence, then, is the simulation of human behavior and 
cognitive processes on a computer. As such, it is also the study of the nature 
of the entire domain of intelligent minds. Artificial intelligence research 


XV 


pRefaCe - xvi 


began in the 1940s and early 1950s. Since then, ai entities have become an 
integral part of modern human life, functioning in a much more sophisti- 
cated way than other common tools. Could these entities become danger- 
ous? Indeed, they already are, as the incident I recounted earlier attests. 

Artificial intelligence and robotics are making their way into everyday 
modern life. And some of the questions I have listed are currently before 
the courts or regulators. But there is little comprehensive analysis in the 
literature about assessing liability for robots, machines, or software that 
exercise varying degrees of autonomy. 

The objective of this book is to develop a comprehensive, general, and 
legally sophisticated theory of the criminal liability for artificial intelli- 
gence and robotics. In addition to the ai entity itself, the theory covers the 
manufacturer, the programmer, the user, and all other entities involved. 
Identifying and selecting analogies from existing principles of criminal 
law, the theory proposes specific ways of thinking through criminal li- 
ability for a diverse array of autonomous technologies in a diverse set of 
reasonable circumstances. 

Chapter 1 contains an investigation of some basic concepts on which 
the development of subsequent chapters is built. I will address two main 
questions: 


1. What can be considered as an artificial intelligence entity? 
2. What are the necessary requirements for the imposition of crim- 
inal liability? 


Because criminal law does not depend on moral accountability, the debate 
about the moral accountability of machines is irrelevant to these ques- 
tions. Although at times there is an overlap between criminal liability, 
certain types of moral accountability, and certain types of ethics, such co- 
incidence is not necessary in order to impose criminal liability. 

The question of the applicability of criminal law to machines involves 
two interrelated, secondary questions: 


1. Is criminal liability applicable to machines? 
2. Is criminal punishment applicable to machines? 


The first question is a core issue of criminal law, and imposition of 
criminal liability depends on meeting its requirements. Only if these re- 
quirements are met does the question of punishability arise, that is, how 
human society can punish machines. The answers proposed in this book 
are affirmative for both questions. Chapters 2 through 5 examine the first 
question, and chapter 6 examines the second. If the affirmative answers 
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offered in this book are accepted, then a new social entity has indeed been 
created, machina sapiens criminalis. The emergence of machina sapiens 
criminalis is examined in chapter 1. Finally, this study answers the ques- 
tion of whether artificial intelligence entities could ever be considered 
legally, if not morally, liable for the commission of criminal offenses, and 
deserving of punishment. 

This book focuses on the criminal liability of A1 entities and does not 
formally venture into the related areas of ethics (including roboethics) and 
morality. Nevertheless, the discussion of criminal liability presented here 
sets the stage for an in-depth consideration of the ethical issues involved, 
which will be necessary given the proliferation of ar entities in everyday 
human life. This book constructs a suitable framework for concrete and 
functional discussion of these issues. 

I wish to thank Dr. Phyllis D. Deutsch of University Press of New En- 
gland for her faith in this project from its very beginning, for guiding the 
publication of the book from inception to conclusion, and for her useful 
comments. I also wish to thank Gabriel Lanyi for his comments, insights, 
and disputations. I thank the anonymous readers for their comments. I also 
thank Ono Academic College and Northeastern University for their gener- 
ous support of this project. Finally, I wish to thank my wife and daughters 
for their staunch support all along the way. 

—G.h. 


The emergence of machina Sapien S criminali § 


1.1. The endless QuesT for Machina sapiens 


1.1.1. history and prehistory of artificial intelligence 

Since the dawn of humanity, we have sought tools to make life easier. In 
the Stone Age, tools were made of stone. As humans discovered the advan- 
tages of metals, metal replaced stone. With the expansion of knowledge, 
tools proliferated and played increasing roles in daily human life. Tools 
were continually challenged by increasingly complicated tasks. When 
they failed, more sophisticated tools were invented; if these succeeded, 
new challenges were posed for them, in a perpetual cycle that continues to 
this day. Mechanical devices have been used to ease daily life since antig- 
uity. Heron of Alexandria, in the first century ap, designed the first steam 
engine and a wind-powered organ, among his many other inventions.’ 
But machines became commonly used in mass production during the first 
industrial revolution, in the eighteenth century. 

The idea of thinking machines evolved together with the insight into 
the human ability to create systematic methods of rational thinking. Des- 
cartes initiated the human guest for such methods in 16372 although he 
did not believe that mechanical devices could achieve reason.’ But Des- 
cartes laid the groundwork for the symbol-processing machines of the 
modern age. In 1651, Hobbes described reason as symbolic calculation.’ 
During the seventeenth century, Leibniz held forth the hope of discovering 
a universal language of mathematics, science, and metaphysics, charac- 
teristica universalis, which would enable replacing thinking with calcula- 
tion, and Pascal designed adding and multiplying machines, most likely 
the first mechanical computers,” operated entirely by humans and inca- 
pable of thinking. Naturally, these machines were not expected to think 
independently. 

The modern idea of a thinking machine is generally associated with 
Ada Lovelace, the daughter of Lord Byron and patroness of Charles Bab- 
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bage, inventor and designer of the analytical engine—the first mechanical, 
programmable computer, but never completed. In 1843, when she was ex- 
posed to Babbage's work, Lady Lovelace asked Babbage whether his ma- 
chine could actually “think.” The notion of mechanical thinking was ex- 
tremely odd in those days, but nevertheless, the guestion itself opened the 
mind for considering the feasibility of unnatural, or “artificial” intelligence. 

But the idea of artificial intelligence could not be examined in practice 
until electricity, and eventually electronic computers, became reality. In 
the 1950s, momentous developments in machine-to-machine translation 
made it possible for computers to communicate with each other, and fur- 
ther developments in human-to-machine translation made it possible for 
human operators to communicate with computers, initially in a limited 
fashion. In time, computer scientists learned to analyze reasoning and rep- 
resent knowledge using symbols to advance the important area of artificial 
intelligence known as knowledge representation.® 

The ability of electronic computers to store large amounts of informa- 
tion and process it at high speeds challenged scientists to build systems 
that could exhibit human capabilities. Since the 1950s, human skills and 
capabilities have increasingly been captured and carried out by electronic 
machines. With the development and popularization of the personal com- 
puter, these capabilities have become affordable for many people. Con- 
tinually increasing memory capacities, speed, reliability, and robustness 
of personal computers are today bringing artificially intelligent tools to 
desktops worldwide. 

During the rapid advances in the field of computers in the 1950s, ar- 
tificial intelligence (a1) developed as a separate area of research that com- 
bined technological investigation with the study of logic and eventually 
of cybernetics, an interdisciplinary field dealing with communication be- 
tween humans and machines, among other issues. Studies in logic in the 
1920s and 1930s helped formalize the methods of reasoning, and pro- 
duced a new form of logic known as propositional and predicate calculus, 
relying on the works of Church, Gédel, Post, Russell, Tarski, Whitehead, 
Kleene, and others.” Developments in psychology, neurology, statistics, 
and mathematics in the 1950s were also incorporated into the growing 
research sphere of AT. 

In the late 1950s, several developments took place that signaled the 
emergence of A1 to the public at large. The most prominent of these was the 
development of chess-playing programs and of the General Problem Solver 
(cpr), designed to solve a wide range of problems, from symbolic integra- 
tion to word puzzles. This brought the abilities of AT to the attention of an 
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enthusiastic public, creating at the same time unrealistic expectations and 
also the first science fiction novels about robots rebelling against humans 
and establishing control over them.° 

AI research proceeded in two main directions: (1) building physical 
devices combined with digital computers, and (2) developing symbolic 
representations. The first effort led to robotics; the second sought to train 
perception to classify certain types of patterns as either similar or dis- 
tinct. A1 research was guided by the assumption that the problem of com- 
monsense knowledge is solvable, and that if humans can solve it, so can 
machines. The problem was formulated as the ability to glean facts in 
situations when not all the information is given, using common sense. For 
many years, this assumption prevented progress in theoretical ai and di- 
verted ai research toward connectionism (neural network models).'° 

By the 1970s, the importance of a1 became apparent to most of the 
world. Governments in developed and developing countries were seek- 
ing long-term commitments of resources to fund intensive research pro- 
grams in A1." Government and private organizations routinely cooperated 
in development programs in the areas of robotics, software, and various 
computer products. These projects were driven by the realization that it 
had become feasible to develop systems that could exhibit such human 
abilities as understanding speech and visual scenes, learning and refining 
knowledge, and making independent decisions. 

Beginning in the 1970s, industry embraced ai technology. Some ai ca- 
pabilities, such as natural language processing, have not yet been fully 
implemented, but in a growing number of fields, ar technology is being 
applied successfully by industry.’ Examples of such areas are biomedical 
microscopy, materials analysis, and robotics. Areas in which a1 has been 
successfully applied have become entirely dependent on this technology 
because common human abilities cannot replace the accuracy, speed, and 
efficiency of these ai tools. 

In the 1980s, ar research made giant gains in the design and develop- 
ment of expert systems in the fields of medicine, finance, and anthropol- 
ogy. The main challenge of expert systems was to develop suitable knowl- 
edge representations in their respective fields. To be readily accessible, 
this knowledge needed to be stored in a form that could be retrieved and 
displayed automatically by the system interfaces, human or other. Many 
expert systems became successful tools, expanded over the years with new 
knowledge and improved with better heuristics.” 

The next challenge was to enable new technologies to be incorporated 
into expert systems shortly after they became available. Development of 
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expert systems resulted in intensive research in the areas of machine learn- 
ing and problem solving, which expanded the use of expert systems into 
many new fields, with development efforts extending beyond the confines 
of academic research, and a1 technology increasingly replacing tradition- 
ally human capabilities.'* With the growing need for ar development in in- 
dustry, the advantages of ar technology were no longer subjects of debate. 

The involvement of industry in ai research grew for three main rea- 
sons. First, the achievements of AT, especially in knowledge engineering, 
were beyond any doubt. Second, advances in hardware made the develop- 
ment of A1 tools more feasible and the products more accessible to users. 
Third, industry had a growing need for faster and better problem-solving 
tools in order to increase productivity. Industry, therefore, adopted the use 
of A1 technology in many areas, such as factory automation, programming, 
office automation, and personal computing. 

The combination of improving abilities in a1 technology, of human 
curiosity, and of industry needs is responsible for the global expansion in 
the use of AT technology. This trend continues into the third millennium, 
as more traditionally human functions are being replaced by ai technolo- 
gies. For example, the South Korean government now uses ai robots as 
soldiers to guard the border with North Korea; as teachers in schools; and 
since 2012, as prison guards.' 

In an eighty-two-page report that outlines the future use of drone air- 
craft, “Unmanned Aircraft Systems Flight Plan 2009—2047,” the US Air 
Force writes that autonomous drone aircraft are key “to increasing effects 
while potentially reducing cost, forward footing, and risk.” The report 
states that, similar to a chess program that can outperform proficient chess 
masters today, future drones will be able to react faster than human pilots 
ever could. But the report also notes potential legal problems: “Increasingly 
humans will no longer be ‘in the loop’ but rather ‘on the loop'— monitoring 
the execution of certain decisions... Authorizing a machine to make lethal 
combat decisions is contingent upon political and military leaders resolv- 
ing legal and ethical questions.”?” 


1.1.2. Defining artificial intelligence, and the endlessness 
of the Quest for machina Sapiens 
Since the beginning of a1, researchers have been trying to develop com- 
puters that actually “think.” This is the holy grail of Ar research.* But to 
develop a thinking machine, it is necessary to first define what exactly 
thinking is. Defining thinking, with regard to both humans and machines, 
proved to be a complicated task for a1 researchers. The development of 
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machines that have the ability to think independently would be an impor- 
tant event for humankind, which has claimed a monopoly over this high 
mental skill. The creation of a true thinking machine would be tantamount 
to the emergence of a new species on earth, the machina sapiens. 

But does human science want to create a new species? Since time im- 
memorial, people have been trying to play God, with the first preventing 
steps mentioned in the Bible.!° Some aspects of modern scientific research 
contain elements of similar conceit, including such endeavors as human 
cloning, biomedical engineering, anti-aging pursuits, and more. The quest 
for the creation of a new species matches this trend. Although creating a 
new species may benefit humans, this is not necessarily the motivation 
behind al research. The reason may be much deeper, touching on the most 
profound of human quests, which, according to the Bible, was prohibited 
to humans following the original sin. 

The first move toward the creation of machina sapiens is to define artifi- 
cial intelligence. Various definitions have been proffered. Bellman defined 
it as “the automation of activities that we associate with human thinking, 
activities such as decision-making, problem solving, learning.”*? Hauge- 
land defined it as “the exciting new effort to make computers think... 
machines with mind, in the full and literal sense.”21 Charniak and McDer- 
mott defined it as “the study of mental faculties through the use of com- 
putational models.””? Schalkoff defined it as “a field of study that seeks to 
explain and emulate intelligent behavior in terms of computational pro- 
cesses.”** Kurzweil defined it as “the art of creating machines that perform 
functions that require intelligence when performed by people.”** Winston 
defined it as “the study of the computations that make it possible to per- 
ceive, reason, and act.”2$ Luger and Stubblefield defined it as “the branch 
of computer science that is concerned with the automation of intelligent 
behavior.”2é Rich and Knight defined it as “the study of how to make com- 
puters do things at which, at the moment, people are better.”27 

At first sight, these definitions seem more confusing than clarifying. 
But according to these and other definitions, AT systems may be grouped 
into four main categories: those that (1) act like humans, (2) think like 
humans, (3) think rationally, and (4) act rationally.” 

AI systems that act like humans are validated by the Turing test, pro- 
posed by the British mathematician in 1950 to provide an operational defi- 
nition of intelligence.” Turing defined intelligent behavior as the ability 
to achieve human-level performance in all cognitive tasks that would pre- 
vent a human interrogator from distinguishing between human and ma- 
chine behavior. The Turing test involves a human “listening” to a conver- 


When roboTS Kill - 6 


sation between a machine and a human. The conversation is carried out in 
writing to avoid identifying the machine through its voice; therefore, the 
“listening” is carried out in reading the text of the guestions and answers. 
The machine is considered to have passed the Turing test if the human 
interrogator cannot tell the difference between the two.*° The Turing test 
assumes egual cognitive abilities for all humans, although conversations 
between a machine and a child, a person with an intellectual disability, a 
very tired person, and the designer of the machine are likely to be signifi- 
cantly different.” 

To identify ai systems that think like humans, it would be necessary to 
define human thinking first. Ar technologies developed as general problem 
solvers were designed to make decisions that are very similar to human 
decisions based on the same information. The modern development of 
cognitive science enabled experimental approaches to mechanical think- 
ing, and to the testing of machine thinking. In another test proposed by 
Turing, the interrogator’s objective is to differentiate between a man and a 
woman in a conversation, a test that depends on the communicative skills 
of the human participants no less than on that of the machine.” 

The effectiveness of the original Turing test has been questioned, espe- 
cially regarding strong ai (in which computers have a mind similar to that 
of humans). Searle, in his famous Chinese Room thought experiment,% 
described a person who does not know Chinese, alone in a room, receiv- 
ing batches of Chinese messages in writing. A rule book, written in the 
person’s native language, allows him or her to look up the Chinese char- 
acters by their shape, and then to answer the messages using this rule 
book. People outside the room, who are conducting the Turing test, are 
convinced that the person inside the room understands Chinese. 

In this thought experiment, the person inside the room simply fol- 
lows the instructions in the book, but neither the person nor the book un- 
derstands Chinese, although both can simulate such understanding. The 
instruction book is, in this case, a computer program. What, then, does it 
mean for humans to understand a foreign language? It is difficult to iden- 
tify an ar system that thinks rationally without defining rationality first. If 
rationality means “right” thinking, it can be represented by formal logic, 
so that given correct information, the machine can aim to produce a com- 
bination of right conclusions. For example, knowing that all tomatoes are 
red and that X is a tomato, the machine should reach the conclusion that 
X is red. Most modern AI systems support formal logic and know how to 
behave in accordance with it. 

Al systems that act rationally are an advanced variation of systems that 
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think rationally. The latter can reach the right conclusions from correct in- 
formation as outsiders, but the former can participate in events, taking the 
right action based on correct information. For example, an A1 goalkeeper 
in a soccer game that sees the ball approaching the goal rapidly can not 
only calculate the speed and angle at which the ball is approaching and 
determine the action required to block it, but also take the required action 
and stop the ball. 

The quest for machina sapiens, however, reaches much deeper than 
these classification and definitions, and if the quest is successful, it will 
answer the question of what makes humans intelligent, and will result in 
the design of intelligent machines. Since the late 1980s, the accepted ap- 
proach has been that intelligent thinking is identified by certain attributes, 
specifically the following five: (1) communication, (2) internal knowledge, 
(3) external knowledge, (4) goal-driven conduct, and (5) creativity.’ Let us 
discuss each of these attributes. 

Communication is considered the most important attribute that defines 
an intelligent entity, because we can communicate with intelligent enti- 
ties. Note that we communicate not only with humans, but with certain 
animals as well, although the range of such communication is narrower 
than our range with humans, and not all ideas can be expressed. For exam- 
ple, you can let your dog know how angry you are, but you cannot explain 
to him the Chinese Room thought experiment. This situation is not differ- 
ent from communicating with a two-year-old human. The more intelligent 
the other entity, the more complex the communication conducted with it. 

Communication assumes proper understanding of the information 
contained in it, and can be used to test the ability to understand compli- 
cated ideas. But communication does not always attest to the quality of 
understanding. Some highly intelligent people, at genius level, are not 
good communicators, and some autistic geniuses cannot communicate at 
all. At the same time, most normative people have advanced communica- 
tion skills, although only a few can communicate about complex matters 
such as Einstein’s general theory of relativity, quantum mechanics, parallel 
universes, or exponential equations. 

Communication can be carried out not only through speech, but by 
other means as well, including writing. Therefore, machines can be quite 
intelligent even if they lack the ability to speak, similar to mute people. So 
although the communication attribute of intelligence is important, there 
are many exceptions to it. The Turing test is based on this attribute, but if 
human communication is so inaccurate, how can we trust it to identify a1? 

Internal knowledge refers to the knowledge of entities about them- 
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selves. This parallels self-awareness. An intelligent entity is supposed to 
know about its own existence, about the fact that it functions in some way, 
about the way in which it integrates into factual reality, and so on. Formal 
logic reasoning showed the way to artificial internal knowledge through 
self-reference." It is possible to program computers to appear that they 
know about themselves, and that they know about knowing about them- 
selves. But many researchers consider this too artificial, and insist that 
it would be difficult to determine whether these computers really know 
about themselves. Yet no alternatives have been suggested to test internal 
knowledge. The question is, how is it possible to know anything about 
another person’s internal knowledge? 

External knowledge refers to factual data about the outside world and 
about factual reality. This attribute is considered crucial in an age when 
knowledge functions as a commodity, especially with relation to expert 
systems.” An intelligent entity is expected to know how to find data about 
the outside world, and to know the facts that make up the factual reality to 
which it is exposed. This attribute assumes memory skills and the ability 
to classify information into seemingly relevant categories. This is the way 
humans assemble their life experience and the way in which they learn. It 
is difficult to act as an intelligent entity if all factual elements are treated 
each time as new. Although factual events are new each time, they do 
contain common characteristics that an intelligent entity should identify. 

For example, medical expert systems, which are designed to diagnose 
diseases based on their symptoms, should be able to identify the common 
characteristics of diseases in many cases, even if these cases vary to an 
extreme degree. An entity that lacks this ability acts in a way similar to 
people who suffer from amnesia. Although they conduct themselves ade- 
quately in a given circumstance, their conduct is not added to their cu- 
mulative experience. This is how simple machines operate: performing a 
certain task, but not knowing that they have done so, and not having the 
ability to draw on this or other experiences to guide them in future tasks. 
So the question is, if inexperienced people can still be considered intel- 
ligent, then why not machines that behave similarly? 

Goal-driven conduct marks the difference between random or arbitrary 
conduct and intended conduct. Goal-driven conduct requires an operative 
plan to achieve the desired goal. For most humans, goal-driven conduct 
is interpreted as intention. If one is hungry and sees an apple, eating it is 
goal-driven conduct, the goal being to appease the hunger. We may say that 
the person ate the apple with the intention of not being hungry anymore. 
Goal-driven conduct is not unique to humans. When a dog sees a juicy 
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bone behind an obstacle, she plans to bypass the obstacle to get the bone. 
Executing the plan is goal-driven conduct on the part of the dog. 

Different creatures may have different goals, of differing levels of com- 
plexity. The more intelligent the entity, the more complex are its goals. 
The goals of dogs are at the level of calling for help when their masters are 
in distress, whereas the goals of humans may include landing on Mars. 
Computers have the ability to plan the landing on Mars, and robots and 
unmanned vehicles are already exploring Mars, searching for data. The 
reductionist approach to goal-driven conduct deconstructs a complex goal 
into many simple ones; achieving these is considered goal-driven conduct. 
Goals and plans for achieving them may be incorporated into computer 
programs. But not all humans pursue complex goals at all times and under 
all circumstances. The question is, what level of complexity of goals is 
needed for a machine to be considered intelligent? 

Creativity involves finding new ways of understanding or doing things. 
All intelligent entities are assumed to have some degree of creativity. When 
a fly tries to get out of the room through a closed window, it crashes against 
the windowpane over and over again, trying invariably the same conduct 
without success. This is the opposite of creativity. At some point, however, 
the fly does seek another way out, which is considered to be more creative. 
A dog may find a way out in much fewer attempts; a human in fewer still. 
Computers may be programmed not to repeat the same conduct more than 
once, and instead to seek other ways of solving the problem. This type of 
behavior is essential for general problem-solving software. 

Creativity is not homogenous, and has degrees and levels. Not all humans 
are considered to be thinking outside the box, and many perform their daily 
tasks exactly the same way day after day. Most lottery players play the same 
numbers time after time for years without winning. What makes their crea- 
tivity different from that of the fly? The fly’s chance of breaking through the 
window is not smaller than that of the lottery player winning the main prize. 
Many factory workers perform the same operations for hours every day, and 
they are considered intelligent entities. The question is, what is the exact 
level of creativity required to be considered intelligent? 

Not all humans who are considered intelligent share all five attributes. 
Why, then, should we use different standards for humans and machines to 
measure intelligence? But each time some new software has succeeded in 
conquering a given attribute, the achievement was rejected for not being 
real communication, internal knowledge, external knowledge, goal-driven 
conduct, or creativity.” Consequently, new tests were proposed to make 
sure that the relevant A1 technology was really intelligent.” 


When roboTs Kill - 10 


Some of these tests have had to do with knowledge representation 
(what the machine knows), decoding (translating knowledge from factual 
reality to its representation), inference (extracting the content of knowl- 
edge), control of combinatorial explosion (preventing endless calculation 
for the same problem), indexing (arranging and classifying knowledge), 
prediction (assessing the probability of possible factual events), dynamic 
modification (self-changing the programs as a result of experience), gener- 
alization (inductive interpretation of factual events), and curiosity (won- 
dering about factual events or seeking reasons for them). 

In their biological sense, all the attributes just mentioned are manifes- 
tations of the human brain. Nobody doubts this fact. These attributes are 
manifestations of neuronal activity in the brain, which can be identified 
and quantified. If so, and if transistors can be functionally activated in the 
same way as neurons, why can these attributes not be reproduced by means 
of transistors?*° The simple answer is that Ar entities are simply not human. 

The process of proposing new tests whenever a given al technology 
succeeds in passing tests devised earlier makes the quest for machina sa- 
piens endless. The reason for this is psychological rather than technologi- 
cal. When talking about AT, we must not envisage regular human beings 
with a robotic-metallic appearance. Yet we will settle for nothing less,“ 
forgetting that artificial intelligence happens to be artificial, not natural. 
When ai technology finally succeeds in passing a certain test, this shows 
us that the problem was not in the technology, but in the test. Because the 
complexity of the human mind is too vast to be tested with simple tests, 
we replace the tests, a process in which we have learned more about the 
human mind, especially about the “bureaucracy” of mind and its inten- 
tionality,” than about technology.” 

The template of the arguments against the definition and feasibility of 
machina sapiens as a machine that possesses truly intelligent personhood 
goes like this: 


1. To possess personhood, an entity must have attribute X. 

2. al technology cannot possess attribute X. 

3. Behavior of a1 technology that is identified to possess attribute 
X demonstrates that it can simulate or imitate that attribute. 

4. Simulation of attribute X is not attribute X itself. 


Therefore, ar technology is not really intelligent. Some scholars called 
this template of arguments a “hollow shell strategy.”** Note that argument 
number 2 and the conclusion result in a catch-22: Ar entities are not intel- 
ligent because A1 technology cannot possess the attribute of intelligence. 
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The contents of attribute X may be made up entirely of advanced in- 
telligence tests, which in this context reflects the belief that intelligence 
can be only human, characterized by the fact that it is exclusively human, 
so that machines cannot perform it. The result is a paradox. Despite re- 
markable advances in ai technology, there has been great frustration with 
the abilities of al, as progress in AI research has underscored how far the 
technology is from imitating the human mind and how much farther the 
mysteries of the human mind still need to be explored. 

In the 1950s, it was difficult to believe that a computer could ever 
defeat humans in chess, and that if it were to happen, computers would 
be considered truly “intelligent.” But in 1997 a computer program (Deep 
Blue) defeated the world champion in chess, and it was still not consid- 
ered intelligent.*® In 2011 another computer program (Watson) defeated 
two of the top Jeopardy players on television, but it was still not consid- 
ered intelligent. Although the computer missed some of the jokes on the 
show, it won the contest. The technology developed for Watson is being 
adapted for use in advanced expert systems, and although it is considered 
good enough to diagnose diseases, many people still do not consider it 
intelligent. 

Al technologies pioneered machine learning, an inductive method of 
learning in which the computer analyzes various cases and situations sub- 
mitted to it, and by means of generalizations, produces an image of the 
facts for future use.” Similar knowledge in humans is called expertise, 
but humans prefer not to refer to these computer systems as intelligent. 
It seems that the creation of machina sapiens recedes further with each 
advance in ar technology, and that ar research has produced an endless 
quest for the new species. 

There are two ways of dealing constructively and positively with the 
endless quest for machina sapiens. The first is through technological re- 
search, seeking ways to reduce the gap between machines and humans. 
This is the path chosen by AT researchers since 1950, following the orig- 
inal quest for machina sapiens in the belief that some day technology 
will be able to imitate the human mind. ar research has had significant 
accomplishments by following this method.** The second way is indus- 
trial. As I will describe, industry is not interested in imitating the human 
mind in all its details. Indeed, industry welcomes the opportunity of using 
Al entities that do not exhibit certain human qualities. So the disadvan- 
tages of machines—the limited ways in which they imitate the human 
mind—become advantages for industry. Turning the disadvantages of the 
machine into advantages for industry makes it possible to increase the use 
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and integration of AT entities in industry. The industrial use of ar technol- 
ogy was the catalyst for the emergence of machina sapiens criminalis. 


1.1.3. Turning Disadvantages into advantages: 
industrial and private Use of artificial intelligence 

Industrial use of artificial intelligence technology is not new. As noted ear- 
lier, ar technology has been embraced in advanced industries since the 
1970s. But whereas in the beginning, a1 technology was used by industry 
because of its similarity to the human mind, later it was used because of its 
differences from the human mind. Industry was quick to understand that 
complete and perfect imitation of the human mind would not be as useful 
as incomplete imitation, so industry encouraged the development of AT 
technology as long as imitation of the human mind was not complete. And 
because complete imitation of the human mind is still far in the future, 
industry and A1 research continue to cooperate. 

How did industry turn a disadvantage into an advantage? Consider the 
calculator. If we enter “1 x 1 =” into a calculator repeatedly, we continue 
to receive the answer “1” invariably, even if we repeat this operation thou- 
sands of times. The process activated by the calculator is the same each 
time. But if we ask a human the same question, we may receive an answer 
the first time—unless the human believes she is being teased —and per- 
haps several more times, but not thousands of times. At some point, the 
human stops answering because she becomes bored, irritated, or nervous, 
or because she loses any desire to answer. 

This is a major problem from the point of view of a1 researchers, and it 
underscores the fact that the human mind may act arbitrarily, sometimes 
for irrational reasons. It would be unthinkable, however, for the calcula- 
tor to refuse to answer “1 x 1 =” even for the thousandth time. For tasks 
of this type, industry prefers entities that are not bored with our requests 
or whims, are not irritated by our questions, and will continue to serve us 
well, no matter how many times we ask the same question. 

It appears that most humans lack these abilities because they have a 
human mind. But machines, which have not succeeded in imitating the 
human mind completely, have the ability to provide us with this kind of 
service. The example just presented is theoretical, of course, as no one 
would really type a thousand times “1 x 1 =” on one’s calculator, a task 
that in itself would require nonhuman skills. But machine skills of this 
nature are required in most industries. 

Consider the customer service representatives of large companies that 
serve hundred of thousands of customers and must provide polite, help- 


The eMergence of Machina sapiens criMinalis - 13 


ful answers to requests of every type. How do these representatives re- 
spond after one call? After a hundred calls? After thousands of calls? How 
does the number of calls affect the quality of service? But the technologi- 
cal inability of a machine to become bored, irritated, or tired is a net ad- 
vantage for industry. An automatic customer-service system services the 
thousandth customer exactly the way it serviced the first one: politely, 
patiently, efficiently, and accurately. 

Expert systems specializing in medical diagnosis are not bored by re- 
peating identical solutions to problems of different patients. Police robots 
are not too frightened to dismantle dangerous explosive devices. Factory 
robots are not bored by performing identical activities thousands of times 
a day. Industry has leveraged the nonhuman qualities of ai entities, turn- 
ing the traditional disadvantages of AT technology (at least from the point 
of view of al researchers) into advantages for modern industry.5 

In practice, these disadvantages are benefitting not only industry. Re- 
search in al technology and industry has made this technology available 
for private consumption as well. Personal robot assistants based on al tech- 
nology are already available, and a1 robots are expected to make their ap- 
pearance in family and private life, including some of the most intimate 
situations. It has already been suggested that “love and sex with robots” 
may provide a better and socially healthier alternative to prostitution.” 
No shame, abuse, or mental or physical harm would result from using the 
Al alternative, and this could result in a real social change. 

Household robots are not offended if they are asked to perform the 
same action over and over, and they do not require vacations, demand 
raises, or ask for favors. Teacher robots are unlikely to teach material out- 
side the curriculum they have been programmed to teach, and prison- 
guard robots are unlikely to be bribed to allow prisoners to escape. These 
nonhuman qualities have made a1 technology highly popular for meeting 
industrial and private needs. 

We can characterize the ar technology required for industrial and pri- 
vate needs as a less than complete and perfect imitation of the human 
mind, but as having some human skills and being capable of an imperfect 
and incomplete imitation of the human mind. The resulting a1 entities are 
not yet machina sapiens, but they possess some human skills for solving 
problems and can imitate some of the capabilities of the human mind. The 
existing capabilities of a1 technology, which are already being used to sat- 
isfy industrial and private needs, are incorporated today in the machina 
sapiens criminalis. 
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1.2. on evolu Tion and devolu Tion: 
Machina sapiens cr iMinalis 


In modern times, technology and law often find themselves on opposite 
sides of the spectrum. Whereas technology serves the cause of the evolu- 
tion of new innovations, the law pulls in the conservative direction. For 
example, technology made human cloning possible in principle, yet in 
most countries, the law prohibited it or initiated a public debate about 
it. What technology considers progress, the law may treat as regress. At 
times the law is inspired by moral conceptions of society, at other times 
by human fears. This is the situation with advanced robotics based on AI 
technology that has produced the machina sapiens criminalis. 


1.2.1. Serve and protect: human f ears of h uman-r obot c oexistence 
Since the beginning of the twenty-first century, reports about research 
in advanced technology have predicted that starting at about the third 
or fourth decade of this century, a new generation of robots will support 
human-robot coexistence.” Under the Fukuma “World Robot Declaration” 
issued in 2004, these robots are anticipated to coexist with humans, assist 
humans both physically and psychologically, and contribute to the realiza- 
tion of a safe and peaceful society.” 

It has been accepted that there are two major types of such robots. 
The first is a new generation of industrial robots capable of manufacturing 
a wide range of products, performing multiple tasks, and working with 
human employees. The second is a new generation of service robots capa- 
ble of performing such tasks as house cleaning, security, nursing, life sup- 
port, and entertainment by coexisting with humans in home and business 
environments. The authors of most published projections on this technol- 
ogy have also evaluated the level of danger to humans and society as a 
result of using these robots.°* Such evaluations provoked a debate about 
the safety of using robots, irrespective of the real level of predicted danger. 

People consider safety only when something is thought dangerous. 
The accelerated technological developments in the area of a1 and robotics 
caused many fears. For example, one of the first reactions to a humanoid 
robot as a caregiver designed to provide nursing care was the fear of hurt- 
ing the assisted human. Would all humans be ready to let their babies and 
children be nursed by robots? Most humans are not experts in technology, 
and most fear what they do not know. 

Further, the vacuum created by the absence of knowledge and certainty 
is sometimes filled by science fiction. In the past, science fiction was rare 
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and had a small following. Today, Hollywood brings science fiction to 
people worldwide. Most blockbusters of the 1980s, 1990s, and twenty-first 
century are classified as science fiction movies, and most of these films ex- 
ploit people’s fears. For example, in 2001: A Space Odyssey (1968), based 
on Arthur C. Clarke’s novel, the central computer of the spaceship is out 
of human control and attempts to assassinate the crew.” Safety is restored, 
after many deaths, only when the computer is shut down.*® 

In The Terminator series of films (The Terminator, 1984; Terminator 
2: Judgment Day, 1991; and Terminator 3: Rise of the Machines, 2003), 
machines are taking over humanity, which is almost extinct. A few sur- 
vivors establish a resistance to oppose the machines, which must be shut 
down to ensure human survival. Even the savior, which happens to be a 
machine, ends up shutting itself down.” In the trilogy of The Matrix (The 
Matrix, 1999; Matrix Reloaded, 2003; and Matrix Revolutions, 2003), ma- 
chines dominate the earth, enslaving humans, to produce energy for their 
own benefit. The machines control the humans’ minds by creating the il- 
lusion of a fictional reality called the “matrix.” Only a few can escape the 
matrix to fight for freedom from the domination of the machines.* In J, 
Robot (2004), based on an Isaac Asimov collection of stories from 1950, an 
advanced model of robots hurts people, one robot is suspected of murder, 
and one of the human heroes is a detective who does not trust robots. The 
overall plot concerns the robots’ attempt to take over humans.5$ 

The influence of Hollywood is vast, so it is reasonable to expect that 
fear of artificial intelligence and robotics will dominate the public mind. 
The more advanced a robot, the more dangerous. A popular topic in sci- 
ence fiction literature and films is revolt and takeover by robots. Thus, 
when people think about advanced robots and technology, in addition to 
utility and unquestionable advantages, they also consider how to protect 
themselves. People will accept the idea of human-robot coexistence only 
if they feel safe from the other, the different, the terrifying: the machine. 

What protection mechanisms can humanity use to ensure the safety of 
coexistence with robots and ai technology?® The first to sound the alarm 
about the dangers of such technology was science fiction literature, which 
was also the first to suggest protection from it. The first circle of protec- 
tion to be proposed was in the realm of ethics focused on safety. The ethi- 
cal issues were presented to the designers and programmers of Ar entities, 
to ensure that built-in software would prevent any unsafe activity of the 
technology.“ One of the pioneers of the attempts to create robot ethics was 
Isaac Asimov. 

Asimov stated three famous “laws” of robotics in his collection of nine 
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science fiction stories assembled under the title J, Robot in 1950: “(1) A 
robot may not injure a human being or, through inaction, allow a human 
being to come to harm; (2) A robot must obey the orders given it by human 
beings, except where such orders would conflict with the First Law; (3) A 
robot must protect its own existence, as long as such protection does not 
conflict with the First or Second Laws.” ® In the 1950s, these laws were 
considered innovative, and restored some confidence in a terrified public. 
After all, harming humans was not allowed. 

Asimov’s first two laws represent a human-centered approach to safety 
in relation to AT robots. They also reflect the general approach that as robots 
gradually take on more intensive and repetitive jobs outside industrial 
factories, it is increasingly important that safety rules support the concept 
of human superiority over robots.® The third law straddles the border be- 
tween human-centered and machine-centered approaches to safety. The 
functional purpose of the robots is to satisfy human needs, and therefore 
robots should protect themselves in order to better perform these tasks, 
functioning as human property. 

But these ethical rules, or laws, were insufficient, ambiguous, and not 
sufficiently broad, as Asimov himself admitted.** For example, a robot in 
police service trying to protect hostages taken by a criminal who intends 
to shoot one of them understands that the only way to stop the murder of 
an innocent person is to shoot the criminal. But the first law prohibits the 
robot from killing or injuring the criminal (by acting), and at the same time 
prohibits it from letting the criminal kill the hostage (through inaction). 
What can we expect the robot to do based on the first law, with no other 
alternatives available? Indeed, what would we expect a human police of- 
ficer to do? Any solution breaches the first law. 

Examining the other two laws does not change the consequences. If 
a human commander ordered the robot to shoot the criminal, the robot 
would still be breaching the first law. Even if the commander were in im- 
mediate danger, it would be impossible for the robot to act. And even if 
the criminal threatened to blow up fifty hostages, the robot would not be 
allowed to protect their lives by injuring the criminal. Matters would be 
even more complicated if the commander were a robot too, but the con- 
sequences would be no different. And the third law would not alter the 
results even if the robot itself were in danger. 

The dilemma of the police robot is not uncommon in a society in 
which humans and robots coexist. Any activity by robots and ar technol- 
ogy in such a society is riddled with such dilemmas. Consider a robot in 
medical service that is required to perform an intrusive emergency surgi- 
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cal procedure intended to save a patient’s life, and to which the patient 
objects. Any action or inaction by the robot breaches Asimov’s first law, 
and any order by a superior is unable to solve the dilemma because an 
order to act causes injury to the patient and an order to refrain from action 
results in the patient’s death. 

Some dilemmas are easier to solve when one of the options involves no 
injury to humans, but this raises other questions. In the previous example 
of robots used as prison guards,® what should such a robot do when a pris- 
oner attempts to escape and the only way to stop the prisoner is by causing 
injury? What should a sex robot do when ordered to commit sadistic acts? 
If the answers are not to act, then the question is, why use such robots in 
first place if their intended actions breach the first law? 

If humanity determines that the destiny of AT robots and other artifi- 
cial entities is to serve and protect humanity in various situations, this 
involves difficult decisions that will have to be made by these entities. 
The terms injury and harm may be wider than specific bodily harm, and 
these entities may harm people in other ways than bodily. Moreover, in 
various situations, causing one sort of harm should be necessary in order 
to prevent greater harm. In most cases, such decision involves complicated 
judgment that exceeds the simple dogmatic rules of ethics. 

The debate about Asimov’s laws gave rise to debate about the moral ac- 
countability of machines.” The moral accountability of robots, machines, 
and al entities is part of the endless quest for machina sapiens. But robots 
exist, they participate in daily human life in industrial and private envi- 
ronments, and they do cause harm from time to time, regardless of whether 
they can be subjected to moral accountability. Therefore, the ethical sphere 
is unsuitable for settling the issue. Ethics involve moral accountability, 
complicated inner judgment, and inapplicable rules. The question, then, 
is what can settle robot functioning, human-robot relations, and human- 
robot coexistence? The answer may lie in criminal law. 


1.2.2. e volution and Devolution: 
machina Sapiens c riminalis as a By-product 
Artificial intelligence technology makes possible a range of industrial and 
private uses of robots. The technology will certainly become more ad- 
vanced in the future as AI research develops over time, and both indus- 
trial and private use of this technology will broaden the range of tasks that 
Al robots will undertake. The more advanced and complicated the tasks, 
the higher the chances of failure in accomplishing them. Failure is a broad 
term that includes various situations in this context. The most common 
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situation is that the task undertaken by the robot has not been accom- 
plished successfully. But some failure situations can involve harm and 
danger to individuals and society. 

For example, the task of prison-guard robots has been defined as pre- 
venting escape by using minimal force against the prisoners. A prisoner 
attempting to escape may be restrained by the robot guard, which holds 
the prisoner firmly but causes injury; the prisoner may then argue that the 
robot has excessively used its power. Analyzing the robot’s actions may 
reveal that it could have chosen a more moderate action, but the robot had 
evaluated the risk as being graver than it actually was. In this case, who is 
responsible for the injury? 

This type of example raises important questions and many arguments 
about the responsibility of the ar entity. If analyzed through the lens of 
ethics, the failure in this situation is that of the programmer or the de- 
signer, as most scientists would argue, not of the robot itself. The robot 
cannot consolidate the necessary moral accountability to be responsible 
for any harm caused by its actions. According to this point of view, only 
humans can consolidate such moral accountability. The robot is nothing 
but a tool in the hands of its programmer, regardless of the quality of its 
software or cognitive abilities. This argument is related to the debate about 
machina sapiens. 

Moral accountability is indeed a highly complex issue, not only for 
machines, but for humans as well. Morality, in general, has no common 
definition that is acceptable in all societies by all individuals. Deonto- 
logical morality (concentrated on the will and conduct) and teleological 
morality (concentrated on the result) are the most acceptable types, and in 
many situations they recommend opposite actions.® The Nazis considered 
themselves deontologically moral, although most societies and individu- 
als disagreed. If morality is so difficult to assess, then moral accountability 
may not be the most appropriate and efficient way of evaluating responsi- 
bility in the type of case we have just examined. 

In this context, the issue of the responsibility of AT entities will always 
return to the debate about the conceptual ability of machines to become 
human-like, so that the endless quest for machina sapiens would become 
an endless guest for a1 accountability. The relevant question here exceeds 
the technological one, and it is mostly a social question. How do we, as a 
human society, choose to evaluate responsibility in situations of harm and 
danger to individuals and society? 

The main social tool available for handling such situations in daily 
life is criminal law, which defines the criminal liability of individuals 
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who harm society or endanger it. Criminal law also has educational social 
value because it educates individuals on how to behave within their so- 
ciety. For example, criminal law prohibits murder; in other words, the 
law defines what is considered to be murder, and prohibits it. This has 
the value of punishing individuals for murder ex post, and prospectively 
educating individuals not to murder ex ante, as part of the rules of living 
together in society. Thus, criminal law plays a dominant role in social 
control.® 

Criminal law is considered the most efficient social measure for di- 
recting individual behavior in any society. It is far from perfect, but under 
modern circumstances it is the most efficient measure. And because it is 
efficient regarding human individuals, then it is reasonable to examine 
whether it is efficient regarding nonhuman entities as well, specifically 
Al entities. Naturally, the first step in evaluating the efficiency of criminal 
law with regard to machines is to examine the applicability of criminal 
law to them. So the question in this context is whether machines may be 
subject to criminal law. 

Because criminal law does not depend on moral accountability, the 
debate about the moral accountability of machines is irrelevant to the 
question at hand. Although at times there is an overlap between criminal 
liability, certain types of moral accountability, and certain types of ethics, 
such coincidence is not necessary in order to impose criminal liability. 
The question of the applicability of criminal law to machines involves two 
interrelated, secondary questions: 


1. Is criminal liability applicable to machines? 
2. Is criminal punishment applicable to machines? 


The first question is a core issue of criminal law, and imposition of crimi- 
nal liability depends on meeting its requirements. Only if these require- 
ments are met does the question of punishability arise, that is, how human 
society can punish machines. The answers proposed in this book are affir- 
mative for both questions. Chapters 2 through 5 examine the first question, 
and chapter 6 examines the second. If the affirmative answers offered in 
this book are accepted, then a new social entity has indeed been created, 
machina sapiens criminalis. 

Machina sapiens criminalis is the inevitable by-product of human ef- 
forts to create machina sapiens. Technological response to the endlessness 
of the quest for machina sapiens has resulted in advanced developments 
in AT technology, which have made it possible to imitate human minds 
and their associated skills better than ever before. Current AT technology 
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is capable of performance that was considered science fiction only a few 
years ago—a situation that has existed over the entire second half of the 
twentieth century, and continues into the twenty-first century. 

Every step along the road of technological development is another step 
in the evolution of machina sapiens. The endless guest for machina sa- 
piens created very high upper and lower thresholds, so high in fact that 
today, it is necessary to be human to be considered machina sapiens. So 
the guest remains endless, and the technological race continues. But crim- 
inal liability does not necessarily require the presence of all human skills. 
To become an offender, one need not use all human skills, regardless of 
whether one possesses such skills. 

Consider the attribute of creativity, discussed earlier.” Naturally, there 
are many types of thieves. Some are deemed creative, others not. When 
examining their criminal liability for a given offense, their human skills of 
creativity are not even considered. We impose criminal liability for theft 
regardless of whether the offender was creative in committing the theft. 
Because creativity is not a condition for the imposition of criminal liability 
on any offender, it is not considered in the legal process. 

When the requirements of criminal liability for the specific offense are 
met, no other qualifications, skills, or thoughts are considered. It is pos- 
sible to argue that although theft does not require creativity, humans are 
assumed to be creative, and therefore criminal liability is applicable to 
them. This type of argument may be relevant in the debate on the endless- 
ness of the quest for machina sapiens, but absolutely not in the question 
of criminal liability. Human skills that are irrelevant for the commission 
of a given offense are not considered as part of the legal process toward 
criminal liability. 

Therefore, whether the offender has been creative in committing the 
offense is irrelevant for the imposition of criminal liability, because there 
is no such requirement for criminal liability. Whether the offender was or 
was not creative outside the commission of the offense is also irrelevant, 
because the legal process focuses only on the facts involved in the com- 
mission of the offense. This is also true of many other attributes consid- 
ered necessary for announcing the birth of machina sapiens. 

It appears, therefore, that in the race to create machina sapiens, signifi- 
cant by-product has been developed: a type of machine that does not have 
the skills to be considered machina sapiens because it cannot fully imitate 
the human mind; but its skills are adequate for various activities in indus- 
trial and private environments. Indeed, the absence of some skills is even 
considered an advantage when focusing on industrial and private use, 
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although AT research considers such absence a disadvantage.” One type of 
activity that this by-product is capable of is the commission of offenses. 

In other words, although the by-product is not capable of many types 
of creative activities, it is perfectly capable of committing offenses. The 
reason for this lies in the definitions and requirements of criminal law for 
the imposition of criminal liability and punishment. These requirements 
are satisfied by capabilities that are far lower than those required to create 
machina sapiens. In the context of criminal law, as long as these require- 
ments are met, there is nothing to prevent the imposition of criminal liabil- 
ity, whether the subject of criminal law is human or not. This is the logic 
that applies to human and nonhuman offenders, such as corporations. 

Thus, as long as all relevant requirements of criminal law are met, a 
new type of subject may be added to the large group of existing subjects to 
criminal law, in addition to human individuals and corporations. These 
subjects may be referred to as machina sapiens criminalis, not a subtype of 
a general machina sapiens, but its by-product. This by-product is consid- 
ered to be less advanced technologically, because it is possible to belong 
to this category of machines without the high-level skills attributed to the 
ideal machina sapiens, if and when achieved. 

From the technological-scientific point of view, any progress toward 
the development of machina sapiens is considered progress (or “evolu- 
tion”), whereas the by-product, machina sapiens criminalis, is more 
of a regress (or “devolution”). Machina sapiens criminalis is a stopping 
point along the race to the top. It is a less-advanced peak, because the re- 
quirements for criminal liability are much lower than those for the ideal 
machina sapiens. Nevertheless, machina sapiens criminalis is not a race 
to the bottom, but merely a stopping point in the race to the top. From 
the criminal law point of view, the technological research may rest at this 
point because the entities to which the criminal law is applicable already 
exist. 

If the range of machines extends between the poles of technically basic 
(“dumb”) machines and machina sapiens, then machina sapiens crimi- 
nalis is situated somewhere in the middle, perhaps closer to the machina 
sapiens pole. But as noted earlier, the emergence of machina sapiens 
criminalis is contingent on a match between the requirements of criminal 
law and the relevant skills and abilities of the machine. Some researchers 
argue that the current law is inadequate for dealing with ar technology, 
and that it is necessary to develop a new legal domain called Robot Law.” 

The argument of this book is that the current criminal law is adequate 
to cope with ai technology. Moreover, if technology were to advance sig- 


When robots Kill - 22 


nificantly toward the creation of machina sapiens, this would make cur- 
rent criminal law even more relevant to addressing ai technology because 
such technology imitates the human mind, and the human mind is already 
subject to current criminal law. Thus, the closer ai technology approaches 
to a complete imitation of the human mind, the more relevant the current 
criminal law becomes. 

Subjecting AT robots to the criminal law may relax our fears of human- 
robot coexistence. Criminal law plays an important role in giving people 
a sense of personal confidence. Each individual knows that all other indi- 
viduals in society are bound to obey the law, especially the criminal law. 
If the law is breached by any individual, society enforces it by means of 
its relevant coercive powers (police, courts, and so on). If any individual 
or group is not subject to the criminal law, the personal confidence of the 
other individuals is severely harmed because those who are not subject to 
the criminal law have no incentive to obey the law. 

The same logic works for humans, corporations, and AT entities.” If any 
of these is not subject to criminal law, the personal confidence of the other 
entities is harmed, and more broadly, the sense of confidence of the entire 
society is harmed. Consequently, society must make all efforts to subject 
all active entities to its criminal law. At times this requires conceptual 
changes in the general insights of society. For example, when the fear of 
corporations that are not subject to criminal law became real, in the seven- 
teenth century, criminal law was applied to them. It is perhaps time to do 
the same for AT entities in order to alleviate human fears.” 

But one more issue must be clarified. There are other creatures living 
among us, in our society, in coexistence with us: animals. Why not apply 
our legal rules concerning animals to A1 entities as well? 


1.2.3. inapplicability of the Zoological 
| egal model to ai Technology 

Not all creatures on earth are considered by humans to be intelligent. Most 
of those considered to be intelligent are mammals. Nevertheless, humans 
and animals have coexisted for many millennia. Since humanity first do- 
mesticated animals for its own benefit, this coexistence has become inten- 
sive. We eat their products (milk, eggs, and so on); feed on their flesh; are 
protected by them (for example, guard dogs); employ them (in agriculture, 
transportation, and so on); and use them for both industrial and private 
needs. Therefore, the law has addressed the issues of our coexistence since 
ancient times.”° 

Examination of both ancient and modern law regarding animals re- 
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veals two important aspects that the law addresses. The first is the law’s 
relation to animals as a property of humans, and the second is our duty 
to show mercy toward animals. The first aspect involves the ownership, 
possession, and other property rights of humans toward animals. For ex- 
ample, if damage is caused by an animal, the human who has the property 
rights over the animal is legally responsible for the damage. If a person is 
attacked by a dog, its owner is legally responsible for any damage. 

In most countries these are issues of tort law, although in some coun- 
tries they are related to criminal law as well, but in either case the legal 
responsibility is the human’s, not the animal’s. Since ancient times, an 
animal was incapacitated only if it was considered too dangerous for so- 
ciety. Incapacitation usually meant killing the animal. This was the case 
if an ox gored a person in ancient times,” and it still is the case if a dog 
bites a person under certain circumstances. No legal system considers an 
animal to be directly subject to the law, especially not to criminal law, 
regardless of the animal’s intelligence. 

The second aspect of law—the duty to show mercy toward animals — 
is aimed at humans. As humans are considered superior to animals 
because of their human intelligence, animals are viewed as helpless. Con- 
sequently, the law prohibits the abuse of power by humans against animals 
because of cruelty. The subjects of these legal provisions are humans, not 
animals. Animals abused by humans have no standing in court, even if 
damaged. The legal “victim” in these cases is society, not the animals, so 
in most cases, these legal provisions are part of criminal law. 

Society indicts humans who abuse animals because such cruelty 
harms society, not the animals. This type of protection of animals differs 
little from the protection of property. Most criminal codes prohibit dam- 
aging another’s property, in order to protect the property rights of the pos- 
sessor or owner. But in the case of cruelty to animals, this type of protec- 
tion has nothing to do with property rights. The legal owner of a dog may 
be indicted for abusing the dog regardless of property rights with respect 
to the dog. These legal provisions, which have existed since time imme- 
morial, form the zoological legal model. The question is why this model 
cannot be applied to a1 entities. 

We are considering three types of entities: humans, animals, and artifi- 
cially intelligent entities. If we wish to subordinate al entities to the crimi- 
nal law of humans, we must be able to justify the resemblance between 
humans and ai entities in this context. Indeed, we should explain why 
Al entities carry more resemblance to humans than to animals. Otherwise, 
the zoological legal model would be satisfactory and adequate for settling 
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the al activity. Thus, the question is, who does an AT robot resemble more: 
a human or a dog? 

The zoological legal model has been examined previously with re- 
spect to al entities when discussing the control of unmanned aircrafts” 
and other machines,” including “new generation robots.”®° For some of 
the legal issues raised in these contexts, the zoological legal model was 
able to provide answers, but the core problems could not be solved by this 
model. When the a1 entity was able to figure out by itself, using its soft- 
ware, how it needed to act, something in the legal responsibility puzzle 
was still missing. Communication of complicated ideas is much easier 
with ar entities than with animals. This is the case regarding external 
knowledge and the quality of reasonable conclusions in various situations. 

An Al entity is programmed by humans to conform to formal human 
logical reasoning. This forms the core reasoning on which the activity 
of the ar entity is based, so that its calculations are accountable through 
formal human logical reasoning. Most animals, in most situations, lack 
this type of reasoning. It is not that animals are not reasonable, but their 
reasonableness is not necessarily based on formal human logic. Emotion- 
ality plays an important role in the activity of most living creatures, both 
animals and humans, and it may supply the drive and motivation to some 
human and animal activity. This is not the case with machines. 

If emotionality is used as a measure, humans and animals are much 
closer to each other than either is to machines. But if pure rationality is 
used as a measure, machines may be closer to humans than to animals. 
Although emotionality and rationality affect each other, the law distin- 
guishes between them regarding their applicability. For the law, especially 
for criminal law, the main factor being considered is rationality, whereas 
emotionality is only rarely taken into account. For example, a murderer’s 
feelings of hatred, envy, or passion are immaterial for his or her convic- 
tion; and only in rare cases, when the provocation defense is used, may 
the court consider the emotional state of the offender.” 

Further, the zoological legal model educates humans to be merciful 
toward animals, as already noted, which is a key consideration under this 
model. In relation to ai entities, however, this consideration is immaterial. 
Because al entities lack basic attributes of emotionality, they cannot be sad- 
dened, made to suffer, disappointed, or tortured in any emotional manner, 
and this aspect of the zoological legal model has no significance regarding 
Al entities. For example, cutting off a dog’s leg for no medical reason is 
considered abuse, and in some countries it is considered an offense. But 
no country considers cutting off a robot’s arm an offense or abuse, even if 
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the robot is a humanoid, and even if the robot is programmed to respond 
to this action “emotionally.” 

Therefore, given that the law prefers rationality over emotionality when 
evaluating legal responsibility, and because the rationality of A1 technology 
is based on formal human logical reasoning, ar technology is much closer 
to humans than to animals from the point of view of the law and with 
regard to all legal aspects. So the zoological legal model is not suitable for 
evaluating the legal responsibility of a1 entities.** To subject al entities to 
the criminal law, it is necessary to use the basic concepts of criminal li- 
ability, which form the general requirements of criminal liability and must 
be met in order to impose criminal liability on any individual (human, 
corporate, or Ar) entity. 


1.3. The Modern offender 


Who is considered an offender under modern criminal law? Most people 
associate the terms criminal and offender with “evil.” Murderers and rap- 
ists are criminals; therefore, they are evil. Within the confines of criminal- 
ity, however, we encounter not only murder and rape, but other behaviors 
as well, which are not considered evil by most people. Some “white- 
collar” crimes are not always considered evil, and many people regard 
them as rather clever ploys or as succumbing to complex bureaucratic 
systems. Similarly, most breaches of traffic laws are not regarded as evil, 
nor are some of the offenses committed under the pressures imposed by 
discharging one’s duty. 

For example, consider the case of a surgeon who conducts a compli- 
cated emergency surgery to save a patient's life, and must work under great 
time pressure, so that the operation fails despite the surgeon’s earnest ef- 
forts, and the patient dies. The post mortem examination reveals that one 
of the surgeon’s acts may be considered negligent, and she becomes crimi- 
nally liable for negligent homicide. Do we consider her evil? 

Indeed, even some of the classic crimes, such as murder, do not always 
appear evil under certain circumstances. For example, a son witnesses the 
daily suffering of his mother, who is dying of a terminal illness; she asks 
him to disconnect her from the cpr machine to end her suffering. If the son 
reluctantly agrees, he is criminally liable for murder.® But is he evil? And 
what if he refuses his mother’s request, in order to watch and presumably 
revel in her suffering? He would not be criminally liable for any offense, 
but would we not consider him evil? 

Countless similar examples could be cited. Evil is not a measure of 
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criminal liability. Morality of any kind and criminal liability are vastly 
different. At times they coincide, but such coincidence is not necessary 
for the imposition of criminal liability. An offender is a person on whom 
criminal liability has been imposed. When the legal requirements of a 
given criminal offense are met by individual behavior, criminal liability 
is imposed, and the individual is considered an offender. Evil may or may 
not be involved. Thus, imposition of criminal liability requires the explo- 
ration of its requirements. 

Criminal law is considered the most efficient means of social control. 
Society, as an abstract body or entity, controls its individual members 
using a variety of measures (for example, moral, economic, or cultural), 
but one of the most efficient is the law. And given that only criminal law 
includes significant sanctions (imprisonment, fines, and even death), it is 
considered the most efficient of the means used to control individuals; this 
is commonly called legal social control. Imposition of criminal liability 
is the application and implementation of legal social control. 

Modern criminal liability is independent of morality of any kind, and 
independent of evil. It is imposed in an organized, almost mathematical 
way. Two cumulative types of requirements must be met for the impo- 
sition of criminal liability: the first has to do with the law, the second 
with the offender. Both types of requirements must be met in order to 
impose criminal liability, and if they are met, no additional conditions 
are required. 


1.3.1. Basic r equirements for a given offense 

The first type of requirement, placed on the law, includes four separate 
requirements that every offense must meet, as defined by law. If the of- 
fense fails to meet even one of these requirements, no court can impose 
criminal liability on individuals for that offense. The four requirements 
are (1) legality, (2) conduct, (3) culpability, and (4) personal liability. Each 
of these four requirements represents a fundamental principle in criminal 
law, that is, the principle of legality, the principle of conduct, the principle 
of culpability, and the principle of personal liability. 

Legality is a requirement for an offense to be considered legal (nullum 
crimen sine lege). It is legality that forms the rules used to determine what 
is criminally “right” and “wrong.” For an offense to be considered legal, 
it must meet the following four cumulative conditions: (1) legitimate legal 
source, (2) applicability in time, (3) applicability in place, and (4) legiti- 
mate interpretation. Only when all four conditions are met is the offense 
considered legal. 
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The offense must have a legitimate legal source that creates and defines 
it.” In most countries, the ultimate legitimate legal source is legislation, 
and case law is not considered a legitimate source because legislation is 
enacted by elected public representatives of society as a whole. Given that 
criminal law exercises legal social control, it should reflect the will of 
society by means of its representatives. The requirement that the offense 
be applicable in time means that retroactive offenses are illegal.“ In plan- 
ning their moves, individuals must know the prohibitions in advance, not 
retroactively. Only in rare cases are retroactive offenses considered legal, 
for example when a new offense is to the benefit of the defendant (it is 
more lenient than its predecessor or allows for a new defense) or when the 
offense covers cogent international custom, or jus cogens, (for example, 
genocide, crimes against humanity, or war crimes).*° 

The offense must also be applicable in place, so that extraterritorial of- 
fenses are illegal.” Criminal law is based on the authority of the sovereign, 
which is domestic, meaning that criminal law must be domestic as well. 
For example, the criminal law of France is applicable in France but not 
in the United States. Thus, extraterritorial offense is illegal (for example, 
French offense that is applicable in the United States). In rare cases, how- 
ever, the sovereign is authorized to protect itself or its inhabitants abroad 
through extraterritorial offenses, as, for example, in the case of the foreign 
terrorists who attacked the US Embassy in Kenya and who may be indicted 
in the United States under US criminal law, although they have never been 
to the United States.” Extraterritorial offenses may also apply in cases of 
international cooperation between states. 

Offenses must be well formulated and phrased. They must be general 
and address an unspecified public (for example, “Samuel Jackson is not 
allowed to...” is not a legitimate offense).°? Offenses must be feasible, 
so that legal social control is realistic (for example, “Whoever breathes 
for more than five minutes uninterruptedly shall be guilty of...” is not a 
legitimate offense). They must also be clear and precise, so that individu- 
als will know exactly what they are allowed to do and what is prohibited 
conduct.” When all these conditions are met, the requirement of legality 
is satisfied. 

Conduct is a requirement that every offense must meet in order to 
be considered legal (nullum crimen sine actu). Modern society prefers 
freedom of thought and has no interest in punishing mere thoughts (cogi- 
tationis poenam nemo patitur). Effective legal social control cannot be 
achieved through mind policing, which is not enforceable. The conduct 
is the objective and external expression of the commission of the offense. 
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Offenses that do not meet this are not legitimate. Throughout legal human 
history, only tyrants and totalitarian regimes have used offenses that lack 
conduct. 

The conduct requirement of some offenses can be satisfied by inac- 
tion. Offenses that criminalize the status of the individual rather than his 
or her conduct are considered status offenses. For example, offenses that 
punish the relatives of traitors simply because they are relatives, regard- 
less of their conduct, are considered status offenses.” So are offenses that 
punish individuals of certain ethnic origin. Most developed countries 
have abolished these offenses, and defendants indicted for such offenses 
are acquitted in court because status offenses contradict the principle of 
conduct in criminal law.” Only when conduct is required may offenses be 
considered legal and legitimate. 

The culpability requirement (nullum crimen sine culpa) must be met 
for an offense to be considered legal. Modern society has no interest in 
punishing accidental, thoughtless, or random events, only events that are 
the result of an individual’s culpability. Someone’s death does not nec- 
essarily indicate the presence of an offender. A person may pass next to 
another exactly when the other person is struck dead by lightning, but 
the passerby is not necessarily culpable. The offense must require some 
level of culpability for the imposition of criminal liability; otherwise, this 
would amount to cruel maltreatment of individuals by society. 

Culpability relates to the mental state of the offender and reflects the 
subjective-internal expression of the commission of the offense. The re- 
quired mental state of the offender, which forms the requirement of culpa- 
bility, may be reflected both in the particular requirements of the specific 
offense and in the general defenses. For example, the offense of man- 
slaughter requires recklessness as its minimal level of culpability. But if 
the offender is insane (general defense of insanity), is an infant (general 
defense of infancy),°° or has acted in self-defense (general defense of self- 
defense),™ no criminal liability is imposed. No culpability can be ascribed 
to such an offender. An offense is considered legal and legitimate only if 
it requires culpability. 

Personal liability is required for an offense to be considered legal. 
Modern society has no interest in punishing one person for the behavior 
of another, regardless of their relationship. Effective legal social control 
cannot be achieved unless all individuals are liable for their own behavior. 
If someone knows that no legal liability is imposed on him for his behav- 
ior, he has no incentive to avoid committing offenses or other types of anti- 
social behavior. Legal social control can be effective only when a person 
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knows that for his own behavior, no other person is liable. Punishment 
deters individuals only if they may be punished personally. 

Personal liability guarantees that each offender would be criminally 
liable and punished only for his or her own behavior. When individuals 
collaborate in committing an offense, each of the accomplices is criminally 
liable only for his or her own part. The accessory is criminally liable for 
accessoryship, and the joint perpetrator for joint perpetration. The various 
types of criminal liability in conjunction with the principle of personal 
liability have established the general forms of complicity in criminal law 
(for example, joint perpetration, perpetration-through-another, conspiracy, 
incitement, and accessoryship). Only when personal liability is required 
can the offense be considered legal and legitimate. 

When all four basic requirements of legality, conduct, culpability, and 
personal liability are met, the offense that embodies them is considered to 
be legitimate and legal. Only then can society impose criminal liability on 
individuals for the commission of these offenses. The legitimacy and legal- 
ity of specific offenses is necessary for the imposition of criminal liability, 
but not sufficient. For the imposition of criminal liability, the particular 
requirements of the offense must be met by the offender as well. These 
requirements are part of the definition of the offense. 


1.3.2. r equirements of a given offense: 
modern criminal | iability as a matrix of minimalism 

The second type of requirement is focused on the offender’s behavior. Each 
offense that meets the requirements to be considered such, determines 
the requirements needed for the imposition of criminal liability based on 
that offense. Although different offenses have differing requirements, the 
formal logic behind all offenses and their structure is similar. The common 
formal logic and structure are significant attributes of modern criminal li- 
ability. In general, these attributes are characterized by minimalism, which 
in this context means that the offense determines only the lower threshold 
for the imposition of criminal liability. In other words, the offender is re- 
quired to meet at least the requirements of the given offense. There are two 
general requirements for an offense: (1) the factual element requirement 
and (2) the mental element requirement. 

The modern structure of the factual element requirement is the same 
in most developed countries. This structure applies the fundamental prin- 
ciple of conduct to criminal liability, and it is identical for every type 
of offense, regardless of its mental element requirement. The factual ele- 
ment requirement is the broad objective-external basis of criminal liability 
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(nullum crimen sine actu), ® designed to answer four main questions about 
the factual aspects of the delinquent event: “What?” “Who?” “When?” and 
“Where?” “What” refers to the substantive facts of the event (what hap- 
pened). “Who” relates to the identity of the offender. “When” addresses 
the time of the event. “Where” specifies the location of the event. 

In some offenses, these questions are answered directly within the 
definition of the offense. In others, some of the questions are answered 
through the applicability of the principle of legality in criminal law.*™ 
For example, the offense “Whoever assaults another person . . .” does not 
relate directly to the questions of “who, 
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when,” and “where,” but the 
questions are answered through the applicability of the principle of legal- 
ity. Because the offense is likely to be general,’ the answer to the “who” 
question is any person who is legally competent. As this type of offense 
may not be applicable retroactively,’ the answer to the “when” ques- 
tion is from the time the offense was validated onward. And because this 
type of offense may not be applicable extraterritorially (with some excep- 
tions), the answer to the “where” question is within the territorial juris- 
diction of the sovereign (with some exceptions). 

Nevertheless, the answer to the “what” question must be incorporated 
directly into the definition of the offense. This question addresses the core 
of the offense and cannot be answered through the principle of legality. 
This approach is at the foundation of the modern structure of the factual 
element requirement, which consists of three main components: conduct, 
circumstances, and results. Conduct is a mandatory component, whereas 
circumstances and results are not. Thus, if the offense is defined as having 
no conduct requirement, it is not legal, and the courts cannot convict indi- 
viduals based on such a charge. Table 1.1 lists the components that answer 
the four questions. 

The conduct component is at the heart of the answer to the “what” 
question. Status offenses, in which the conduct component is absent, are 
considered illegal, and in general they are abolished when discovered.1% 
But the absence of circumstances or of results in the definition of an of- 
fense does not invalidate the offense. These components are aimed at 
meeting the factual element requirement with greater accuracy than by 
conduct alone. Therefore, there are four formulas that can satisfy the fac- 
tual element requirement: 


1. conduct 

2. conduct + circumstances 

3. conduct + results 

4. conduct + circumstances + results 
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Table 1.1 - Target Questions of the Factual Element Components 


fac TUal elemen Tcomponen TS QUe STion S 

conduct “what” 

circumstances “what,” “who,” “when,” “where” 
results “what” 


The modern structure of the mental element requirement has been 
widely accepted in most developed countries. This structure applies the 
fundamental principle of culpability in criminal law (nullum crimen sine 
culpa). The principle of culpability has two main aspects: positive and 
negative. The positive aspect (what needs to be in the offender’s mind in 
order to impose criminal liability) relates to the mental element, whereas 
the negative aspect (what should not be in the offender’s mind in order to 
impose criminal liability) relates to the general defenses." For example, 
imposition of criminal liability for manslaughter requires recklessness as 
mental element, but it also requires that the offender not be insane. Reck- 
lessness is part of the positive aspect of culpability, and the general de- 
fense of insanity is part of its negative aspect. 

The positive aspect of culpability in criminal law concerns the in- 
volvement of mental processes in the commission of the offense. The 
mental element has two important aspects: cognition and volition. Cogni- 
tion is the individual’s awareness of the factual reality. In some countries, 
awareness is called knowledge, but in this context there is no substantive 
difference between awareness and knowledge, both of which can relate 
to data from the present or the past, but not from the future.’ A person 
may assess or predict what will be in the future, but not know or be aware 
of it. Prophetic skills are not required for criminal liability. Cognition in 
criminal law refers to a binary situation: the offender either is aware of 
fact X or is not. Partial awareness is not accepted in criminal law, and it is 
classified as unawareness. 

Volition has to do with the individual’s will, and it is not subject to 
factual reality. An individual may wish for unrealistic events to occur 
or to have occurred, in the past, the present, or the future. Volition is 
not binary because there are different levels of will, the three basic ones 
being positive (A wants X), neutral (A is indifferent toward X), and nega- 
tive (A does not want X). The cognitive and volitive aspects combine 
to form the mental element requirement, as derived from the positive 
aspect of culpability in criminal law. In most developed countries, there 
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are three principal forms of mental element, which are differentiated 
based on the cognitive aspect. The three forms represent three layers of 
positive culpability: (1) mens rea (general intent), (2) negligence, and 
(3) strict liability. 

The highest layer of the mental element is that of mens rea, which 
requires full cognition. The offender is required to be fully aware of the 
factual reality. This layer involves examination of the offender’s subjective 
mind. Negligence, the second layer, is cognitive omission: the offender is 
not required to be aware of the factual element, although based on objec- 
tive characteristics, she could and should have had awareness of it. Strict 
liability is the lowest layer of the mental element; it replaces what was 
formerly known as absolute liability. Strict liability is a relative legal pre- 
sumption of negligence based on the factual situation alone, which may be 
refuted by the offender (praesumptio juris tantum). Cognition relates the 
factual reality, as just noted. The relevant factual reality in criminal law is 
that which is reflected by the components of the factual element. 

From the perpetrator’s point of view, however, when committing the 
offense, only the conduct and circumstance components of the factual 
element exist in the present; the results occur in the future. Cognition is 
restricted to the past and the present, and therefore it relates to conduct 
and circumstances. Although results occur in the future, the possibility of 
their occurrence ensuing from the relevant conduct exists in the present, 
so that cognition can also relate to the possibility of the occurrence of the 
results. For example, in the case of homicide, A aims a gun at B and pulls 
the trigger. At this point, A is aware of his conduct, of the existing circum- 
stances, and of the possibility of B’s death as a result of his conduct. 

Volition is considered immaterial for both negligence and strict liabil- 
ity, and can be added only to the mental element requirement of mens rea, 
which embraces all three basic levels of will. Table 1.2 compares the gene- 
ral requirements of the three layers, or forms, of mental element. 

Because in most legal systems, the default requirement for the mental 
element is mens rea, negligence and strict liability offenses must specify 
explicitly the relevant requirement. The explicit requirement may be listed 
as part of the definition of the offense or included in the explicit legal tra- 
dition of interpretation. If no explicit requirement of this type is men- 
tioned, the offense is classified as a mens rea offense, which is the default 
requirement. The relevant requirement may be met not only by the same 
form of mental element, but also by a higher-level form. Thus, the mental 
element requirement of the offense is the minimal level of mental element 
needed to impose criminal liability; a lower level is insufficient for im- 
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Table 1.2 - Schematic Comparison of the Requirements 
of Mental Element Forms 


re QUiremen T 


Form Cognition Volition 
Mens rea subjective cognition only in result offenses 
(because of the relevant 
level) 
Negligence cognitive omission none 
Strict liability presumed cognitive none 
omission 


Table 1.3 » Schematic Comparison of the Requirement 
for Satisfying the Mental Element Forms 


men Tal elemen Tre QUiremen Tof... iS SaTiSfie DThro Ugh ... 
mens rea negligence mens rea negligence or 
strict liability mens rea strict liability or 
negligence or 
mens rea 


posing criminal liability for the offense. Table 1.3 compares schematically 
the requirements for satisfying the three forms of mental element. 

This structure of criminal liability forms a matrix of minimalist re- 
quirements. Each offense embodies the minimum requirement for the im- 
position of criminal liability, and meeting these requirements is sufficient 
to impose criminal liability. No additional psychological elements are re- 
quired, so that the modern offender is an individual who meets the mini- 
mal requirement of an offense. The offender is not required to be “evil,” 
“mean,” or “wicked” —only to meet all the requirements of the offense. 
The imposition of criminal liability is therefore dry and rational (as op- 
posed to emotional), and resembles mathematics. 

Thus, the matrix of minimalist requirements for criminal liability has 
two aspects: structural and substantive. For example, if the mental ele- 
ment of the offense requires only awareness, then no other component of 
the mental element is required (structural aspect), and the required aware- 
ness is defined by criminal law irrespective of its meaning in psychol- 
ogy, philosophy, theology, and so on (substantive aspect). The substantive 
aspect is discussed in detail in chapters 2, 3, and 4. 
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Nevertheless, it is clear that the structure of criminal liability has 
been designed for humans, with the abilities of humans—and not other 
creatures—in mind. The mental element requirement relies on the human 
spirit, soul, and mind. Can machines, therefore, be examined based on 
human standards of spirit, soul, and mind? Considering these insights, 
how can criminal liability be imposed on entities lacking in spirit and 
soul? 

It is important to remember, however, that although insights in crimi- 
nal liability rely on the human spirit and soul, the imposition of criminal 
liability itself does not depend on these terms that carry deep psychologi- 
cal meaning. If an entity, any entity, meets both the factual and mental ele- 
ment requirements of the offense, then criminal liability may be imposed 
with or without spirit, with or without soul. This understanding is not 
new, and perhaps it is not highly innovative in the twenty-first century. 
A similar understanding was reached in the seventeenth century, long 
before the emergence of modern ai technology, but at a time when other 
nonhuman creatures committed offenses and it was necessary to subject 
them to criminal law. These creatures have neither spirit nor soul, but they 
have the ability to be subjects of criminal liability, which is being imposed 
on them to this day. These creatures are corporations. 


1.3.3. The case of nonhuman corporations (r ound 1) 

Since the seventeenth century, corporations have been among the modern 
offenders, although they lack spirit, soul, and a physical body.'!? Corpo- 
rations were recognized already by Roman law, but the evolution of the 
modern corporation began in the fourteenth century, when English law 
demanded permission from the king or Parliament to recognize a cor- 
poration as legal." Early corporations in the Middle Ages were mostly 
ecclesiastical bodies that were active in the organization of church prop- 
erty. From these bodies evolved associations, commercial guilds, and 
professional guilds that eventually formed the basis for commercial 
corporations. 

During the sixteenth and seventeenth centuries, hospitals and univer- 
sities were also commonly incorporated.“* In addition to these, commer- 
cial corporations developed to solve problems of division of ownership 
among several owners of a business.“ When people created a new busi- 
ness, ownership could be divided between them by establishing a corpo- 
ration and dividing the shares and stocks among the shareholders. This 
pattern of ownership division has been perceived as efficient in minimiz- 


The eMerge nce of Machina sapiens criMinalis - 35 


ing the risks of the owners regarding the financial hazards of the business, 
and as a result, corporations have become common." 

The use of corporations flourished during the first industrial revolu- 
tion, to the point where they have been identified both with the fruits of 
the revolution and with the misery of the lower classes and the workers 
(who were created by the revolution). Corporations were regarded as being 
responsible for the poverty of the workers, who did not share in the profits, 
and for the continuing abuse of children employed by the corporations. 
As the industrial revolution progressed, public and social pressure on the 
corporations increased, until legislators were forced to restrict the activi- 
ties of corporations. 

In the beginning of the eighteenth century, the British Parliament en- 
acted statutes against the abuse of power by corporations, the very power 
granted to them by the state for the benefit of social welfare.1!® To ensure 
the effectiveness of these statutes, they included criminal offenses. The 
offense most often used for this purpose was that of public nuisance.’ 
This legislative trend deepened as the revolution progressed, and in the 
nineteenth century, most developed countries had advanced legislation 
regarding the activities of corporations in a variety of contexts. To be 
effective, this legislation included criminal offenses as well, prompt- 
ing the conceptual question, how can criminal liability be imposed on 
corporations? 

Criminal liability requires a factual element, whereas corporations pos- 
sess no physical body. Criminal liability also requires a mental element, 
whereas corporations have no mind, brain, spirit, or soul. Some Euro- 
pean countries refused to impose criminal liability on nonhuman crea- 
tures, and revived the Roman rule whereby corporations were not subject 
to criminal liability (societas delinquere non potest). But this approach 
was highly problematic because it created legal shelters for offenders. For 
example, an individual who did not pay taxes was criminally liable, but 
when the individual was a corporation, it was exempt. This provided an 
incentive for individuals to work through corporations and evade paying 
taxes. Eventually, all countries subjected corporations to criminal law, but 
not until the twentieth century. 

The Anglo-American legal tradition accepted the idea of the criminal 
liability of corporations early because of its many social advantages and 
benefits. In 1635, criminal liability for corporations was enacted and im- 
posed for the first time. This was a relatively primitive imposition of 
criminal liability because it relied on vicarious liability, but it enabled 
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the courts to impose criminal liability on corporations in a way that was 
separate from the criminal liability of any of its owners, workers, or share- 
holders. This structure remained in effect throughout the eighteenth and 
nineteenth centuries.’ 

The major disadvantage of criminal liability based on vicarious liabil- 
ity was that it required valid vicarious relations between the corporation 
and another entity, which in most cases happened to be human.” Con- 
sequently, when the human entity acted without permission (ultra vires), 
the corporation was exempt of responsibility. To exempt a corporation of 
liability, it was sufficient to include a general provision in its incorpora- 
tion papers prohibiting the commission of any criminal offense on behalf 
of the corporation.‘ This deficiency resulted in the replacement in Anglo- 
American legal systems of the model of criminal liability for corporations 
during the late nineteenth century and early twentieth century. 

The new model was based on the identity theory. In some types of cases, 
the criminal liability of corporations derives from its organs, whereas in 
other types of cases, criminal liability is independent. When the criminal 
offense requires an omission (for example, not paying taxes, not fulfilling 
legal requirements, or not observing workers’ rights), and the duty to act 
is the corporation’s, the corporation is criminally liable independently, re- 
gardless of any criminal liability of other entities, whether human or not. 
When the criminal offense requires an act, the acts of its organs are related 
to the corporation if they were committed on its behalf, with or without 
permission.” The same structure works for the mental element, for mens 
rea, negligence, and strict liability alike.’ 

Consequently, the criminal liability of the corporation is direct, not 
vicarious or indirect.” If all requirements of an offense are met by the 
corporation, it is indicted regardless of any proceedings that may be in 
effect against any human entity. If convicted, the corporation is punished 
separately from any human entity. Punishments for corporations are con- 
sidered to be not less effective than for humans. To complete this picture, 
we will return to discuss the case of nonhuman corporations (Round 2) in 
relation to the question of the sentencing of AT entities. 

However, the main significance of the modern legal structure of crimi- 
nal liability of corporations is conceptual. Since the seventeenth century, 
criminal liability has not been a uniquely human domain. Other entities, 
nonhuman, are also subject to the criminal law. Some adjustments were 
needed to make this legal structure applicable, but eventually, nonhuman 
corporations have become subject to criminal law. Today, this seems en- 
tirely natural, as it should. Given that the first barrier was crossed in the 
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seventeenth century, the road may be open to crossing the next barrier that 
stands before the imposition of criminal liability on AT entities. 

In the following chapters, the criminal liability of ar entities is ana- 
lyzed in detail for all types of offenses. Because the major obstacle on the 
way to criminal liability for AT entities is that of the mental element, the of- 
fenses are divided according to their mental element requirements (mens 
rea, negligence, and strict liability). Next, the applicability of general de- 
fenses for al entities is analyzed. Finally, in the last chapter, the applicabil- 
ity of human punishments to AT entities is discussed. 


ai criminal Iia Bili Ty for inTen Tional offen SeS 


In 1981, a thirty-seven-year-old Japanese employee in a motorcycle factory 
was killed by an AT robot working near him. The robot identified the em- 
ployee as a threat to its mission and calculated that the most efficient way 
to eliminate the threat was to push the worker into an adjacent machine. 
Using its very powerful hydraulic arm, the robot smashed the surprised 
worker into the operating machine, killing him instantly, after which it re- 
sumed its duties without further interference.’ The legal question is, who 
is to be held criminally liable for this homicide? 

Intentional offenses make up most of the criminal offenses in most 
countries. To impose criminal liability in these cases, both factual element 
and mental element requirements must be met by the apparent offender. 
Consequently, the question of the criminal liability of ar entities for in- 
tentional offenses is whether ai entities have the capability to meet these 
requirements accumulatively. Note, however, that even if AT entities have 
such capability, this does not exempt humans from their criminal liability, 
if humans are involved in the commission of the offense. 

To explore these issues, we examine the capability of ar entities to 
meet the requirements of both factual and mental elements of intentional 
offenses to allow the imposition of human criminal liability for intentional 
Al offenses. 


2.1. The facTual eleMenT r eQuire MenT 


The factual element requirement structure (actus reus) is identical for all 
types of offenses: intentional, negligence, and strict liability, as previously 
noted.’ This structure contains one mandatory component (conduct) for 
all offenses, and two optional components (circumstances and results) for 
some offenses. Let us now discuss the capability of machines to meet the 
factual element requirement considering this structure. 
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2.1.1. conduct 

Conduct, as already noted, is the only mandatory component of the factual 
element reguirement; in other words, an offense that does not reguire con- 
duct is illegal and illegitimate, and no criminal liability can be legitimately 
imposed based on it. In independent offenses, the conduct component of 
the factual element requirement may be expressed both by an act and by 
omission. In derivative criminal liability, the conduct component may be 
expressed also by inaction, with some restrictions. We will examine these 
forms of conduct in relation to the capabilities of ar machines. 

In criminal law, act is defined as material performance with a factual- 
external presentation. According to this definition, the materiality of the 
act is manifest in its factual-external presentation, which differentiates 
the act from subjective-internal matters related to the mental element. 
Because thoughts have no factual-external presentation, they are related 
not to the factual, but to the mental element. Will may initiate acts, but 
in itself has no factual-external presentation, and is therefore considered 
part of the mental element. Consequently, involuntary and unwilled ac- 
tions, as well as reflexes, are still considered acts as far as the factual ele- 
ment requirement is concerned.* 

But although unwilled acts or reflexes are considered acts, criminal 
liability is not necessarily imposed for such offenses because of reasons 
involving the mental element requirement or general defenses. For ex- 
ample, B physically pushes A in the direction of C. Although the physical 
contact between A and C is involuntary for both, it is still considered an 
act. But it is unlikely that criminal liability for assault will be imposed on 
A, because the mental element requirement for assault has not been met. 
Acts committed as a result of loss of self-control are still considered acts, 
but loss of self-control is a general defense, which if proven and accepted, 
exempts the offender from criminal liability. 

This definition of an act concentrates on its factual aspects to the ex- 
clusion of mental aspects.* The definition is also sufficiently broad to in- 
clude actions originating in telekinesis, psychokinesis, and so on, if these 
are possible,” as long as they have a factual-external presentation. If act 
were restricted to “willed muscular construction” or “willed bodily move- 
ment,”” this would prevent imposition of criminal liability in cases of 
perpetration-through-another (for example, B in the example just cited) 
because no act has been performed, and it would be possible to assault 
someone and be exempt from criminal liability, by pushing an innocent 
person upon the victim. 
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Conseguently, criminal law considers as an act any material perfor- 
mance with a factual-external presentation, whether willed or not. On 
this basis, ar technology is capable of performing acts that satisfy the con- 
duct requirement. This is true not only for strong a1 technology, but also 
for much lower technologies. When a robot moves its arms or any other 
of its devices, it is considered to act. This is true when the movement is 
the result of inner calculations carried out by the robot, but not only then. 
Even if the robot is fully actuated by a human operator through remote 
control, any movement of the robot is considered an act. 

Therefore, even non—A1 technology robots have the factual capabil- 
ity of performing acts, regardless of the motives or reasons for the act. 
This does not necessarily mean that these robots are criminally liable for 
their acts, because the imposition of criminal liability must also satisfy 
the mental element requirement. But as far as performing an act in order 
to satisfy the conduct component requirement is concerned, any mate- 
rial performance with a factual-external presentation is considered an act, 
whether the physical performer is a strong al entity or not. 

In criminal law, omission is defined as inaction contradicting a legiti- 
mate duty to act. According to this definition, the term legitimate duty is 
of great significance. The opposite of action is not omission, but inaction. 
If doing something is an act, then not doing anything is inaction. Omission 
is an intermediate degree of conduct between action and inaction. Omis- 
sion is not mere inaction, but is inaction that contradicts a legitimate duty 
to act: the omitting offender is required to act, but fails to do so. If no act 
has been committed, but no duty to act is imposed, no omission has been 
committed.’ 

The omitting offender is, therefore, punished for doing nothing in spe- 
cific situations where he or she should have done something owing to a 
certain legitimate duty. For example, in most countries parents have a legal 
duty to care for their children. In these countries, the breach of this duty 
may form an offense. In this situation, the parent is not punished for acting 
in a wrong way, but for not acting although having a legal duty to act in a 
specific way. The requirement to act must be legitimate in the given legal 
system, and in most legal systems, the legitimate duty may be imposed 
both by law (legal duty) and by contract (contractual duty). *° 

The modern concept of conduct in criminal law acknowledges no sub- 
stantive or functional differences between acts and omissions for the im- 
position of criminal liability." Therefore, any offense may be committed 
both by act and by omission. Socially and legally, commission of offenses 
by omission is no less severe than commission by act.’* Most legal systems 
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accept this modern approach, and there is no need to explicitly reguire 
omission to be part of the factual element of the offense. The offense de- 
fines the prohibited conduct, which may be committed both through acts 
and through omissions." 

On this basis, a1 technology can perform omissions that satisfy the 
conduct requirement. This is true not only for strong ar technology, but 
also for much lower technologies. Physically, commission by omission 
requires doing nothing. There is no doubt that any machine is capable of 
doing nothing; therefore, any machine is physically capable of committing 
an omission. Naturally, for the inaction to be considered omission, there 
must be a legal duty that contradicts the inaction. If such duty exists, origi- 
nating in law or contract, and the duty is addressed to the machine, then 
there is no question that the machine is capable of committing an omission 
with respect to that duty. 

This is also the situation regarding inaction. Inaction is the complete 
factual opposite of an act. If an act is doing something, then inaction 
is not doing it, or doing nothing. Whereas omission is inaction contra- 
dicting a legitimate duty to act, inaction requires no such contradiction. 
Omission is not doing when there is an obligation to do, whereas inac- 
tion is not doing when there is no obligation to do anything. Inaction 
is accepted as a legitimate form of conduct only in derivative criminal 
liability (for example, attempt, joint perpetration, perpetration-through- 
another, incitement, or accessoryship), and not in complete and inde- 
pendent offenses. 

In these instances, when inaction is accepted as a legitimate form of 
conduct, it is physically committed in the same way as omission. Con- 
sequently, if ar technology is capable of commission through omission 
of conduct, it is also capable of commission through inaction. This does 
not necessarily mean that machines or robots are automatically criminally 
liable for these omissions or inactions, because to impose criminal liabil- 
ity, it is necessary to also meet the mental element requirement, and it is 
not sufficient to satisfy the factual element requirement. 

Thus, the mandatory component of the factual element requirement 
(conduct) can be satisfied by machines. These machines need not be 
highly sophisticated or even based on A1 technology. Simple machines are 
capable of conduct under the definitions and requirements of criminal law. 
For the imposition of criminal liability on any entity, this is an essential 
step, even if not a sufficient one. No criminal liability can be imposed if 
the conduct requirement is not satisfied, but conduct alone is not sufficient 
for the imposition of criminal liability. 
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2.1.2. circumstances 

Circumstances are not a mandatory component of the factual element re- 
quirement. Some offenses require circumstances in addition to conduct, 
and some do not. Circumstances are defined as factual components that 
describe the conduct but do not derive from it. According to this defini- 
tion, circumstances specify the criminal conduct in more accurate terms. 
When defining specific offenses, circumstances are required especially 
when the conduct component is too broad or vague, and greater specific- 
ity is needed in order to avoid over-criminalization of situations that are 
considered legal by society. 

In the case of most offenses, the circumstances represent the factual 
data that make the conduct criminal. For example, in most legal systems, 
the conduct in the offense of rape is having sexual intercourse (although 
the specific verbs may vary). But in itself, having sexual intercourse is not 
an offense, and it becomes one only if it is committed without consent. 
The factual element of “without consent” is what makes the conduct of 
“having sexual intercourse” criminal. In this offense, the factual compo- 
nent “without consent” functions as a circumstance. Additionally, the 
factual component “with a woman” also functions as circumstance be- 
cause it describes the conduct (raping a chair is not an offense). 

In this example, the factual element of the offense of rape is “having 
sexual intercourse with a woman without consent.” Whereas “having 
sexual intercourse” is the mandatory conduct component, “with a woman” 
and “without consent” function as circumstance components that de- 
scribe the conduct more accurately, to make sure that the factual element 
requirement of rape is specified adequately, in a way that avoids over- 
criminalization. 

According to the definition, circumstances are not derived from the 
conduct in order to allow distinguishing them from the result compo- 
nent.” For example, in homicide offenses, the conduct is required to cause 
the “death” of a “human being.” Death describes the conduct and also de- 
rives from it because it is the conduct that caused death. Therefore, in ho- 
micide offenses, “death” does not function as a circumstance. The factual 
data that functions as a circumstance is “human being.” The victim was a 
human being long before the conduct took place, and therefore it does not 
derive from the conduct. But these words also describe the conduct (caus- 
ing the death of a human being, not of an insect), and therefore function 
as a circumstance. 

On this basis, AT technology is capable of satisfying the circumstance 
component requirement of the factual element. When the circumstances 
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are external to the offender, the identity of the offender is immaterial, and 
therefore the offender may be human or a machine, without affecting the 
circumstances. In the earlier example of rape, the circumstance “with a 
woman” is external to the offender. The victim is required to be a woman 
regardless of the identity of the rapist. The raped woman is still considered 
“woman” whether she was attacked by a human or a machine, or has not 
been attacked at all. 

When the circumstances are not external to the offender but are re- 
lated somehow to the offender’s conduct, the circumstances assimilate 
within the conduct component. In the example of rape, the circumstance 
“without consent” describes the conduct as if it were part of the conduct 
(how exactly did the offender have sexual intercourse with the victim). 
To satisfy the requirement of this type of circumstance, the offender must 
commit the conduct in a particular way. Consequently, meeting the re- 
quirement of this type of circumstance is the same as committing the con- 
duct. As already discussed, it is within the capabilities of machines to 
meet the conduct component requirement. 


2.1.3. r esults and c ausal connection 

Results are not a mandatory component of the factual element reguire- 
ment. Some offenses reguire results in addition to conduct, and some do 
not. Results are defined as a factual component that derives from the con- 
duct. According to this definition, results specify the criminal conduct 
with greater accuracy. Results are defined as deriving from the conduct in 
order to allow distinguishing them from the circumstances. For example, 
in homicide offenses, the conduct is reguired to cause the “death” of a 
“human being.” The death describes the conduct and also derives from it, 
because the conduct caused the death. Therefore, in homicide offenses, 
“death” does not function as a circumstance, but as a result. '8 

In the context of the factual element, the results derive from the con- 
duct through factual causal connection. Although additional conditions 
exist for this derivation (legal causal connection), the factual reguirement 
is a factual causal connection. Conseguently, proof of results reguires 
proving a factual causal connection.’® The factual causal connection is 
defined as a derivative connection, in which, were it not for the conduct, 
the results would not have occurred the way they have. According to this 
definition, the results are the ultimate consequences of the conduct, that 
is, causa sine qua non.” 

The factual causal connection relates not only to the mere occurrence 
of the results, but also to the way in which they occurred. For example, A 
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hit B with her car, and B died. Because B was terminally ill, A may argue 
that B would have died anyway in the near future, so B’s death is not the 
ultimate result of A’s conduct, and the result would have occurred anyway, 
even without A’s conduct. But because the factual causal connection has 
to do with the way in which the results occurred, the requirement is met 
in this example: B would not have died the way he did had A not run 
him over. 

On this basis, ar technology is capable of satisfying the result com- 
ponent requirement of the factual element. To achieve the results, the of- 
fender must initiate the conduct. Commission of the conduct forms the 
results, and the results are examined objectively to determine whether 
they are derived from the commission of the conduct.”! Thus, when the 
offender commits the conduct, and the conduct has been carried out, it is 
the conduct, not the offender, that is the cause of the results, if there was 
a result. The offender is not required to cause, separately, any results, but 
only to commit the conduct. Although the offender begins the factual pro- 
cess that forms the results, this process is initiated only through the com- 
mission of the conduct component.” 

Because a1 technology is capable of committing conducts of all types, 
in the context of criminal law it is capable of causing results deriving from 
this conduct. For example, when a robot operates a firearm and causes it 
to shoot a bullet toward a human individual, this fulfills the conduct com- 
ponent requirement of homicide offenses. At that point, a causal connec- 
tion test is used to examine the conduct and determine whether it caused 
that individual’s death. If it did, both the result and conduct component 
requirements are met, although physically the robot “did” nothing but 
commit the conduct. Because meeting the conduct component require- 
ment is within the capabilities of machines, as already discussed, so is 
meeting the results component requirements. 


2.2. The MenTal eleMenT r eQuire MenT 


The imposition of criminal liability for intentional offenses requires the 
fulfillment of both factual and mental element requirements. The mental 
element requirement of intentional offenses is general intent, more accu- 
rately known as mens rea, meaning “evil mind” in Latin. Because the term 
general intent can at times be confused with intent or specific intent, we 
prefer to use the more accurate term, mens rea. If A1 technology is capable 
of fulfilling the mens rea requirement, it is possible, feasible, and achiev- 
able to impose criminal liability on ai entities for intentional offenses. 
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First, we explore the structure of the mens rea reguirement, then we turn 
to the fulfillment of the requirement by a1 technology. 


2.2.1. Structure of the mens r ear equirement 
In the modern age, mens rea expresses basic types of mental elements, as 
it embodies the idea of culpability most effectively. This is the only mental 
element that combines cognition and volition. The mens rea requirement 
expresses the internal-subjective relation of the offender with the factual 
commission of the offense.” In most legal systems, the mens rea require- 
ment functions as the default option of the mental element requirement. 
Therefore, unless negligence or strict liability are explicitly required as 
mental elements of the offense, mens rea is the required mental element. 

This default option is also known as the presumption of mens rea.” 
Accordingly, all offenses are presumed to require mens rea, unless there 
is some explicit deviation from this requirement. Because mens rea is the 
highest level of known mental element requirement, this presumption has 
great significance. Indeed, most offenses in criminal law require mens rea 
and not negligence or strict liability. None of the mental element compo- 
nents, including mens rea components, is independent or stands alone for 
itself. For example, the dominant component of mens rea is awareness. If 
we require the offender to be aware, the question arises, aware of what? 
Awareness cannot stand alone because it would be meaningless. 

Consequently, all mental element components must relate to facts or 
to some factual reality. The relevant factual aspect for criminal liability 
consists, naturally, of the factual element components (conduct, circum- 
stances, and results), as discussed earlier.” Factual reality contains many 
more facts than the components of factual element, but all other facts are 
irrelevant for the imposition of criminal liability. For instance, in the rape 
example, the relevant facts are “having sexual intercourse with a woman 
without consent,”’® and the rapist must be aware of these facts. Whether 
the offender was or was not aware of other facts as well (for example, the 
color of the woman’s eyes, the fact that she was pregnant, or her suffering) 
is immaterial for the imposition of criminal liability. 

Thus, for the imposition of criminal liability, the object of the mental 
element requirement is the factual element components. Naturally, this 
object is much narrower than the complete factual reality, but the factual 
element represents the decision of society regarding what is relevant for 
criminal liability and what is not. Nevertheless, other facts and the mental 
relation to them may affect the punishment, but they are immaterial for 
the imposition of criminal liability. For example, a particularly cruel rapist 
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would be convicted of rape, whether he was cruel or not, but his punish- 
ment would likely be much harsher than that of a less cruel rapist. 

Identifying the factual element components as the object of mens rea 
components is the basis for the structure of mens rea. Mens rea has two 
layers of requirement: cognition and volition. The layer of cognition con- 
sists of awareness. The term knowledge is also used to describe the cogni- 
tion layer, but awareness seems more accurate. Both awareness and knowl- 
edge function the same way and have the same meaning in this context. 
We can be aware only of facts that occurred in the past or are occurring at 
present, but we cannot be aware of future facts. 

For example, we can be aware of the fact that A ate an ice-cream cone 
two minutes ago, and we can be aware of the fact that B is eating one right 
now. If C said that she intends to eat an ice-cream cone, we can predict it, 
foresee it, or estimate the probability that it will happen, but we cannot 
be aware of it, simply because it has not occurred yet. If the criminal law 
had required offenders to be aware of future facts, it would have required 
prophetic skills. As far as we know, most criminals do not possess such 
skills and are far from being prophets. 

In this context, the offender’s temporal point of view determines the 
point at which the conduct is actually performed. Therefore, from the of- 
fender’s point of view, the conduct component always occurs in the pres- 
ent. Consequently, awareness is a relevant component of mens rea in re- 
lation to conduct. Circumstance is defined as factual data that describes 
the conduct but does not derive from it.” To describe current conduct, 
circumstances must exist in the present as well. For example, the circum- 
stance “with a woman” in the offense of rape, described earlier, must exist 
simultaneously with the conduct “having sexual intercourse.” 

The raped person must be a woman during the commission of the of- 
fense for the circumstance requirement to be fulfilled. Therefore, aware- 
ness is a relevant component of mens rea in relation to circumstances as 
well. The situation is different, however, in relation to results, which are 
defined as a factual component that derives from the conduct; this means 
that results must occur after the conduct.” For example, B dies at 10:00:00 
after which, at 10:00:10, A shoots her. In this case, the conduct (the shoot- 
ing) is not the cause of the other factual event (B’s death), which does not 
function as a result. 

From the offender’s point of view, given that the results occur after the 
conduct, and because the offender’s temporal point of view determines 
the point at which the conduct is performed, the results occur in the 
future. Consequently, because results do not occur in the present, aware- 
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ness is not relevant for them. The offender is not expected to be aware of 
the results, which have still not occurred from his point of view. But al- 
though the offender cannot be aware of the future results, he is capable of 
predicting them and assessing the probability of their occurrence. These 
capabilities are present simultaneously with the actual performance of 
the conduct. 

For example, A shoots B. At the point in time when the shot is fired, 
B’s death has not yet occurred, but in the act of shooting, A is aware of the 
possibility of B’s death as a result of the shot. Therefore, awareness is a rel- 
evant component of mens rea in relation to the possibility that the results 
will occur, but not in relation to the results themselves. Awareness of this 
possibility is not required to be related to how reasonable or probable the 
occurrence of the result is. If the offender is aware of the mere existence of 
the possibility, whether its probability is high or low, that the result will 
follow from the conduct, this component of mens rea is present.” 

The additional layer of mens rea is that of volition. This layer is ad- 
ditional to cognition and based on it. Volition is never alone, but always 
accompanies awareness. Volition relates to the offender’s will regarding 
the results of the factual event. In rare offenses, volition may relate to mo- 
tives and purposes beyond the specific factual event, and is expressed as 
specific intent.” The main question regarding volition (apart from the of- 
fender’s awareness of the possibility that the results will follow from the 
conduct) is whether the offender wanted the results to occur. Because from 
the offender’s point of view the results occur in the future, they are the 
only reasonable object of volition. 

From the offender’s temporal point of view, neither circumstances nor 
conduct have anything to do with will. The raped person was a woman 
before, during, and after the rape, regardless of the rapist’s will. The sexual 
intercourse is what it is at that point in time, regardless of the rapist’s will. 
If the offender argues that the conduct occurred against his will—that is, 
the offender did not control it—this argument is related to the general de- 
fense of loss of self-control, which we will discuss in a moment.*! Thus, as 
far as conduct and circumstances are concerned, no volition component is 
required, only awareness. 

In the factual reality there are many levels of will, but criminal law 
accepts only three: (1) intent (and specific intent), (2) indifference, and (3) 
rashness. The first represents positive will (the offender wants the results 
to occur), the second represents nullity (the offender is indifferent about 
the results), and the third represents negative will (the offender does not 
want the results to occur, but has taken unreasonable risks that caused 
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them to occur). For example, in homicide offenses, at the moment when 
the conduct is committed, if the offender 


1. wants the victim to die, it is intent (or specific intent); 

2. is indifferent as to whether the victim lives or dies, it is indif- 
ference; 

3. does not want the victim to die, but assumes unreasonable risk 
in this regard, it is rashness. 


Intent is the highest level of will accepted by criminal law. In most 
countries, intended homicide is considered murder. Indifference is the 
intermediate level, and rashness is the lowest level of will. Both indif- 
ference and rashness are known as recklessness. In most countries, reck- 
less homicide is considered manslaughter. Consequently, if the offense 
requires recklessness, this requirement can be fulfilled by proof of intent 
because a higher level of will satisfies lower levels. But if the offense re- 
quires intent or specific intent, this requirement can be fulfilled only by 
intent or specific intent. 

It is easier to sum up the structure of mens rea if we divide offenses 
into conduct offenses and result offenses. Conduct offenses are those 
whose factual element requires no results; result offenses are those whose 
factual element requires results.” This division clarifies the structure of 
mens rea because volition is required only in relation to results. Therefore, 
results require both cognition and volition, whereas conduct and circum- 
stances require only cognition. The structure of mens rea can be described 
schematically as shown in table 2.1. 

Therefore, in conduct offenses, whose factual element requirement 
contains conduct and circumstances, the mens rea requirement contains 
awareness of these components. In result offenses, whose factual element 
requirement contains conduct, circumstances, and results, the mens rea 
requirement contains awareness of the conduct, of the circumstances, and 
of the possibility that the results will occur. In addition, in relation to re- 
sults, the mens rea requirement contains intent or recklessness, according 
to the definition of the offense. 

This general structure of mens rea is a template that contains terms de- 
rived from the terminology of the sciences of the mind (awareness, intent, 
recklessness, and so on). To examine whether a1 entities are capable of ful- 
filling the mens rea requirement of given offenses, the definition of these 
terms must be explored. Next we will discuss separately the cognitive and 
volitive aspects of mens rea. 
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Table 2.1 - General Structure of Mens Rea 


men Srea re QUiremen T 


Type of offen Se 
Cognition Volition 


Conduct offense awareness of conduct — 
+ 


awareness of circumstances intent 
awareness of conduct 


+ 
Result offense awareness of circumstances or 
+ 


awareness of the possibility | recklessness 
of the occurrence of results (indifference/ 
rashness) 


2.2.2. f ulfillment of the cognitive aspect 

The cognitive aspect of mens rea contains awareness, and therefore the 
relevant question is whether ar technology is capable of consolidating 
awareness. Because the term awareness may have different meanings in 
different scientific spheres (for example, psychology, theology, or law), 
to answer this question we must examine the legal meaning of this term; 
specifically, we must examine awareness based on its legal definition in 
criminal law. Even if the legal meaning may differ from other meanings 
of the term, given that our discussion relates to criminal liability, only the 
legal meaning is relevant. 

In criminal law, awareness is defined as perception of factual data by 
the senses, and its understanding. The roots of this definition lie in the 
psychological understandings of the late nineteenth century. Previously, 
awareness was equated with consciousness, that is, the physiological- 
bodily state the mind is in when the human is awake. By the end of the 
nineteenth century, psychologists argued that awareness involves a much 
more complicated situation of the human mind.™ In the 1960s, the modern 
understanding of the human mind began to consolidate toward the con- 
cept of the sum total of internal and external stimulations that the indi- 
vidual is aware of at a given point in time. 

Consequently, it became apparent that the human mind is not constant 
but dynamic and regularly changing, and could be described as a flow of 
feelings, thoughts, and emotions (“stream of consciousness”).*° It was also 
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understood that the human mind is selective, meaning that humans are ca- 
pable of focusing their minds on certain stimuli while ignoring others, so 
that the ignored stimuli do not enter the human mind at all. If the human 
mind were to include all internal and external stimuli and had to pay at- 
tention to each one of them, it would be incapable of functioning. 

The function of the human and animal sensory systems is to absorb 
the stimuli (light, sound, heat, pressure, and so on) and transfer them to 
the brain for the processing of factual information. Processing is executed 
in the brain internally until a relevant general image of the factual data is 
created. This process is called perception, which is considered one of the 
basic skills of the human mind. 

Many stimuli are active at any one time. To enable the creation of an 
organized image of factual data, the human brain must focus on some of 
the stimuli and ignore others. This is accomplished through the process of 
attention, which permits the brain to concentrate on some stimuli while 
ignoring others. In practice, the other stimuli are not entirely ignored, 
but they exist in the background of the perception process. The nervous 
system still retains adequate vigilance that allows it to absorb other stimuli 
when the process of attention is at work. 

For example, Archie is watching television and is focused on it. Edith 
calls him because she wants to speak with him. When she first calls, he 
does not react. When she calls him for the second time, much louder, he 
reacts, asks her what she wants, and says that he must go to the bathroom. 
When Archie is focused on the television, many of the stimuli in his envi- 
ronment are ignored (for example, the sound of his heartbeats, the smells 
from the kitchen, and the pressure in his bladder), allowing him to focus 
on the television. But his nervous system is still sufficiently alert to absorb 
some of these stimuli. Edith’s first call is just another sound to be ignored 
by the process of attention. But when she calls the second time, the at- 
tention process used to focus on the television is stopped, and the other 
stimuli are absorbed and receive some attention. This is why Archie sud- 
denly remembers that he must go to bathroom. 

Perception includes not only absorbing stimuli, but also processing 
them into a relevant general image, which is what usually gives meaning 
to the accumulation of stimuli. Processing the factual data into a rele- 
vant general image is accomplished through unconscious inference, so 
that there is absolutely no need for awareness to carry this out.’ But the 
result of the processing—that is, the relevant general image—is a con- 
scious result. Thus, although the human mind is not conscious of most of 
the process, it is conscious of its results when the relevant general image 
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is accepted. Conseguently, the human mind is considered to be aware only 
when the relevant general image is accepted. 

This process forms the essence of human awareness, which is the final 
stage of perception. Perception of factual data by the senses, and its un- 
derstanding, results in the creation of a relevant general image, which is 
the awareness of the factual data. For example, it is not the eyes but the 
brain that is the sight organ of humans. Eyes function as mere sensors that 
deliver factual data to the brain. Only when the brain creates the relevant 
general image is the human considered to be aware of the relevant sight. 
Consequently, a human in a vegetative situation, with fully functioning 
eyes, is not considered to be seeing (or to be aware of what she sees), 
unless the sights are combined into a relevant general image. 

As a result, for us to consider a human to be aware of certain factual 
data, two accumulative conditions are required: (1) absorbing the factual 
data by senses, and (2) creating a relevant general image about these data 
in the brain. If one of these conditions is missing, the person is not consid- 
ered to be aware. Awareness is a binary issue: the person either is aware or 
is not. Partial awareness is meaningless. An offender may be aware of part 
of the factual data, that is, fully aware of some of the data, but not partly 
aware of a certain fact. If the facts were not absorbed, or if no relevant 
general image has been created, the offender is considered to be unaware. 

At times, the term knowledge is used to describe the cognitive aspect 
of mens rea, as already noted. Is there a difference between knowledge 
and awareness? When examined functionally, there is no difference be- 
tween these terms as far as criminal liability is concerned because they 
both refer to the same idea of cognition.*” Moreover, knowledge has been 
explicitly defined as awareness at times. But it seems that within the 
context of the mental element, the more accurate term is awareness rather 
than knowledge. 

Outside of the criminal law context, awareness is more related to con- 
sciousness than is knowledge. Knowledge is also related to the cognitive 
process, but it refers more to information than consciousness does. Knowl- 
edge represents perhaps a deeper cognitive process than awareness, but 
within the specific context of the mental element requirement in criminal 
law, awareness seems to be the more accurate term. 

In court, proving the full awareness of the offender beyond any rea- 
sonable doubt, as required in criminal law, is not an easy task. Awareness 
relates to internal processes of the mind, which do not necessarily have 
external expressions. Therefore, criminal law has developed evidential 
substitutes for this task. These substitutes are presumptions, which in cer- 
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tain types of situations presume the existence of awareness. Two major 
presumptions are recognized in most legal systems: (1) the willful blind- 
ness presumption, as a substitute for awareness of conduct and of circum- 
stances; and (2) the awareness presumption, as a substitute for awareness 
of the possibility of the occurrence of the results. Before exploring these 
presumptions and their relevance to a1 technology, we must examine how 
Al technology meets the cognitive aspect of the mens rea requirement. 

Is an Al robot capable of being aware of conduct, circumstances, or the 
possibility of the occurrence of the results in the context of criminal law?*? 
The process of awareness may be divided into two stages, as noted. The 
first stage consists of absorbing the factual data by means of the senses. 
At this stage, the primary role is played by the devices used to absorb the 
factual data. The human devices are organs, such as the eyes (for sensing 
light), the ears (for sensing sound), and so on. These organs absorb the fac- 
tual data (such as sights, light, sounds, pressure, and texture) and transfer 
it to the brain for processing. 

Al technology has the capability and the relevant devices to absorb 
any factual data that can be collected by the five human senses. Cameras 
absorb light and transfer the data to processors,*° microphones pick up 
sounds,*! transducers sense pressures, thermometers sense temperature, 
hygrometers sense humidity, and so on. Indeed, advanced technologies are 
capable of much more accurate sensing than their corresponding human 
organs. For example, cameras may absorb light waves at frequencies that 
the human eye cannot detect, and microphones can sense sound waves 
that are inaudible to human ears. 

How many of us can guess the exact outside temperature and humidity 
within 0.1 degrees? What about 0.0001 degrees? And what about relative 
humidity? Most of us cannot. By contrast, even the simplest electronic 
sensors can perform accurate measurements of these data and transfer the 
information to the appropriate processors. Ar technology, therefore, is ca- 
pable of meeting the requirements of the first stage of awareness, indeed 
much better than humans. 

The second stage of the awareness process consists of creating a rele- 
vant general image about these data in the brain, that is, full perception. 
Naturally, most ar technologies, robots, and computers do not possess bio- 
logical brains, but they possess artificial ones. Most of these “brains” are 
embodied in the hardware (processors, memory, and so on) of the technol- 
ogy they use. Are these brains capable of creating a relevant general image 
out of the factual data they absorb? Humans create the relevant general 
image by analyzing the factual data, which enables them to use the infor- 
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mation, transfer it, integrate it with other information, and act based on 
it—in other words, to understand it.” 

Consider the example of security robots based on ar technology. The 
task of these robots is to identify intruders and either alert human troops 
(police, army) or stop the intruders on their own. Their sensors (cameras, 
microphones) absorb the factual information and transfer it to the pro- 
cessors, which are then supposed to identify the intruder. To do so, the 
processors analyze the factual data, making sure not to mistake police and 
other security personnel in the area for intruders. The analysis is based on 
identifying and discriminating between changing sights and sounds. The 
robots may compare the shape and color of clothes and use whatever other 
attributes they need to make the identification. 

Following this short process, the robot assesses the probabilities. If 
the probabilities do not produce an accurate identification, the robot ini- 
tiates a process of vocal identification and asks the suspicious figure to 
identify itself, provide a password, or perform any other action relevant to 
the situation. If the intruder answers, the sound is compared with sound 
patterns stored in memory. After all the necessary factual data has been 
collected, the robot has sufficient information to make a decision, on the 
basis of which it can act. Indeed, this robot has created a relevant general 
image out of the factual data absorbed by its sensors, which it was able to 
use, transfer, and integrate with other information, and then act based on 
it—in other words, to understand it. 

Consider how a human guard would act in this situation: very likely, 
in the same way. The human guard would identify a figure or a sound 
as suspicious, whereas the robot would examine a change in the current 
image or sound. We prefer robot guards because they work more thor- 
oughly than their human counterparts and do not fall asleep on the job. 
Both the human guard and the robot use their memories to try to identify 
the figure or the sound by comparing it with patterns stored in memory. 
But whereas the robot cannot forget figures, humans can. The human 
guard is not always sure, but the robot assesses all the probabilities. The 
human guard calls out to ask the figure to identify itself, and so does the 
robot. Both the human and the robot compare the answer with information 
stored in memory — yet another function that the robot carries out more 
accurately than its human counterpart. Following all the data collection, 
both human and robot guards make a decision. They both understand the 
situation. 

We say that the human guard was aware of the relevant factual data. 
Can we not say the same about the robot guard? There is no reason why 
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not. Its internal processes are more or less the same as those of the human 
guard, except that the robot is more accurate, faster, and works more thor- 
oughly than the human guard. When they become aware of a suspicious 
figure or sound, they both absorb the data and act accordingly. 

It is possible to argue that the human guard may have absorbed much 
more factual information than the robot—for example, smells—and 
that a human is capable of filtering out irrelevant information, such as 
cricket sounds. This argument does not invalidate the analysis just pre- 
sented. Robots are also capable of absorbing additional factual data, such 
as smells. But as already noted, humans use the attention process to focus 
on a portion of the factual data, as Archie did in the example cited earlier. 
Although humans are capable of absorbing a wide range of factual data, 
this may prove to be a distraction for attending to the task at hand. Robots 
may be programmed to ignore such data. 

If smells are considered irrelevant for a task of guarding, then robots 
will not consider this data. If they were human, we would say that the 
robots will not pay attention to this data. Humans filter out irrelevant data 
using the process of attention that runs in the background of the human 
mind. Robots, however, examine all factual data and eliminate the irrele- 
vant options only after analyzing the data thoroughly. We may now ask 
ourselves which type of guard we humans would prefer: one who uncon- 
sciously does not pay attention to factual data, or one who examines all 
factual data thoroughly? 

In sum, al technology is also capable of fulfilling the requirements for 
the second stage of awareness. And given that the two stages already de- 
scribed are the only stages of the awareness process recognized by crimi- 
nal law, we may conclude that a1 technology is capable of meeting the 
awareness requirement in criminal law. 

Nevertheless, some of us may have the feeling that something is still 
missing before we conclude that machines are capable of awareness. This 
may be correct if we consider awareness in its broader sense, as it is used 
in psychology, philosophy, or the cognitive sciences. But in this book 
we are examining the criminal liability of ar entities, and not the wide 
meaning of cognition in psychology, philosophy, the cognitive sciences, 
and so on. Therefore, the only standards of awareness that are relevant 
here are the standards of criminal law. The other standards are irrele- 
vant for the assessment of criminal liability imposed on either humans or 
Al entities. 

The criminal law definition of awareness is indeed much narrower 
than corresponding definitions in the other spheres of knowledge. This is 
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true for the imposition of criminal liability not only on ai entities, but also 
on humans. The definitions of awareness in psychology, philosophy, or 
the cognitive sciences may be relevant for the endless quest for machina 
sapiens, but not for machina sapiens criminalis, which is based on the 
definitions of criminal law.“ This distinction is one of the best illustra- 
tions of the difference between machina sapiens and machina sapiens 
criminalis. 

As noted earlier, awareness itself is very difficult to prove in court, es- 
pecially in criminal cases where it must be proved beyond any reasonable 
doubt.“ Therefore, criminal law developed two evidential substitutes for 
awareness: the presumptions of willful blindness and awareness. Accord- 
ing to the willful blindness presumption, the offender is presumed to be 
aware of the conduct and circumstances if he or she suspected that they 
existed but did not verify that suspicion.* The rationale of this presump- 
tion is that because the “blindness” of the offender to the facts is willful, 
he or she is considered to be aware of these facts even if not actually aware 
of them. 

If the offender wished to avoid the commission of the offense, he or 
she would have checked the facts. For example, the rapist who suspects 
that the woman does not consent to having sexual intercourse with him 
predicts that if he were to ask her, she would refuse, in which case there 
would be no doubt about her lack of consent. Therefore, he ignores his 
suspicion, and at his interrogation says that he thought she consented be- 
cause he heard no objection. The willful blindness presumption equates 
unchecked suspicion with full awareness of the relevant conduct or cir- 
cumstances. The question is whether this presumption is relevant to A1 
technology. 

According to the awareness presumption, a person is presumed to be 
aware of the possibility of the occurrence of the natural results of his or her 
conduct.” The rationale of this presumption is that humans have the basic 
skills necessary to asses the natural consequences of their conduct. For 
example, when firing a bullet at someone’s head, the shooter is presumed 
to be able to asses the possibility of death as a natural consequence of the 
shooting. Humans who lack such skills, permanently or temporarily, have 
an opportunity to refute the presumption. The question is whether this 
presumption is relevant to AT technology. 

Awareness is indeed difficult to prove for humans. But because within 
a criminal law context, the processes that comprise awareness can be mon- 
itored accurately by ar technology, there is no need for any substitute. In 
human terms, this would be the same as if the brain of every individual 
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were constantly being scanned and every thought recorded. If awareness 
of factual data could be identified on such a brain scan, proving awareness 
beyond any reasonable doubt would be a simple task. 

Given that every act of an al entity can be monitored and recorded, in- 
cluding all processes that comprise awareness within a context of criminal 
law, it is entirely achievable to prove the awareness of an AT entity regard- 
ing particular items of factual data. Therefore, it is not necessary to prove 
the awareness of an al entity by means of substitutes, because awareness 
can be proven directly. But if it were necessary to prove awareness using 
substitutes, this could also be accomplished. Strong Ar technologies use 
algorithms to assess probabilities in order to make reasonable decisions. 
A minimum probability defines the feasibility, probability, or “reasonable- 
ness” of every event. This praxis can be applied beneficially to awareness 
substitutes. 

For humans, only a realistic suspicion is considered adequate for will- 
ful blindness. Thus, if the probability of a particular factual event occur- 
ring is above a certain rate and the A1 entity ignores the possibility, this 
may be considered willful blindness. The reason a human would not check 
such a suspicion is immaterial for the applicability of the willful blindness 
presumption, and the same is true for al entities. So if the argument is that 
the al entity ignores the possibility out of some evil or concealed desire, 
as humans may do, the reason for this ignorance is immaterial, just as it is 
in the case of humans. 

As noted earlier, evil is not one of the components of criminal liabil- 
ity. Thus, the reason for ignoring a suspicion (that is, an alternative with 
a probability higher than a certain threshold) is immaterial, regardless of 
whether the motive is evil. The natural consequence of certain conduct 
has a reasonable probability of occurring. A machine can assess reason- 
ability quantitatively by weighing the relevant circumstances. Strong AI 
technology is capable of identifying the options that humans would call 
“natural.” Thus, although proving the awareness of an AT entity is not sub- 
ject to the difficulties of proving human awareness, and therefore there is 
no need to use the awareness substitutes, these substitutes can be used to 
prove the awareness of AT entities. 


2.2.3. f ulfillment of the Volitive aspect 
The volitive aspect of mens rea contains three levels of will: intent, indif- 
ference, and rashness. As already noted, indifference and rashness are 
commonly referred to as recklessness. The question is whether ar technol- 
ogy is capable of consolidating these levels of will. Because these terms 
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have different meanings in various scientific spheres (psychology, theol- 
ogy, philosophy, law, and so on), to answer this guestion we must exam- 
ine the legal meaning of these terms; specifically, their legal definitions in 
criminal law. Even if the legal meaning of these terms differs from other 
meanings, given that the guestion relates to criminal liability, only the 
legal meaning is relevant. 

Intent is the highest level of will accepted by criminal law. There is 
some confusion about the meanings of the terms intent, general intent, and 
specific intent.* General intent is a common term for mens rea. To avoid 
confusion with intent, we use the term mens rea and not general intent. 
The term intent refers to the highest level of will embodied in the volitive 
aspect of mens rea. The term specific intent refers also to the highest level 
of will. Intent and specific intent have identical levels of will. But intent 
refers to will toward the result components of the factual element, whereas 
specific intent refers to motives and purposes, not to results.*° Specific 
intent is relatively rare. 

Purpose is factual data that are supposed to derive from the conduct. 
The mission of the conduct (or its destiny) is to achieve the purpose. Pur- 
pose offenses do not require that the purpose actually be achieved, only 
that the offender intends to achieve the purpose. For example, “Whoever 
says anything for the purpose of intimidating . . .” is an offense that does 
not require actual intimidation of anyone, only the presence of such pur- 
pose. Motive consists of internal-subjective feeling, and its satisfaction is 
the conduct derived from it. For example, in “Whoever performs X out of 
hatred . . .” the conduct X satisfies the hatred. 

Unless purpose and motive are explicitly required by the offense, they 
are immaterial for the imposition of criminal liability.” For example, (1) 
A killed B out of hatred, and (2) A killed B out of mercy. These are both 
considered murders because the offense of murder does not require spe- 
cific intent to achieve a set purpose or any motives. But if the offense had 
explicitly required specific intent toward a set purpose or any motives, 
proving the specific intent would have been a condition for the imposi- 
tion of criminal liability in that offense. Again, the requirement of specific 
intent is relatively rare. 

Given that the only difference between intent and specific intent in- 
volves their objects (results versus motives and purposes), and that the 
level of will of both is identical, the following analysis of intent applies to 
specific intent as well. Intent is defined as “aware will” that accompanies 
the commission of conduct, the results of which will occur. In the defini- 
tion of specific intent, results are replaced by motives and purposes as fol- 
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lows: aware will that accompanies the commission of conduct, the motive 
for which is satisfied or the purpose of which is achieved. 

Intent is an expression of positive will, that is, the will to make a fac- 
tual event occur.*? Although there are higher levels of will than intent (for 
example, lust, longing, or desire), in criminal law intent has been accepted 
as the highest level of will required for the imposition of criminal liabil- 
ity.°? Consequently, no offense requires a higher level of will than intent, 
and if intent is proven, it satisfies all other levels of will, including reck- 
lessness (indifference and rashness). 

We must distinguish between aware will and unaware will. Unaware 
will is an internal urge, impulse, or instinct of which the person is not 
aware and which is naturally uncontrollable. Individuals cannot control 
their will unless they are aware of it. Being aware of one’s will does not 
guarantee the ability of controlling it; but controlling the will requires 
awareness of it and the activation of conscious processes in the mind to 
cause the relevant activity to start, cease, or continue without interference. 
Imposing criminal liability based on intent requires an individual who is 
aware of his will so that he may control it. 

Intent does not require some abstract aware will. The aware will must 
be focused on certain targets: results, motives, or purposes. Intent is an 
aware will focused on these targets. For example, the murderer’s intent is 
an aware will focused on causing the victim’s death. For the intent to be 
relevant to the imposition of criminal liability, this will must exist simul- 
taneously with the commission of the conduct and must accompany it. If 
a person kills another by mistake, and then, after the victim’s death, devel- 
ops the will to kill the victim, this is not intent. Only when the murder is 
committed simultaneously with the existence of the relevant will can the 
will be considered intent. 

Proving intent is much more difficult than proving awareness. Al- 
though both are internal processes of the human mind, intent relates to 
future factual situations, and awareness relates to current facts. Aware- 
ness is rational and realistic, but intent is not necessarily so. For example, 
a person may have the intention of becoming a fish, but she cannot have 
an awareness of being a fish. Because of the serious difficulties in proving 
intent, criminal law developed evidentiary substitutes for it. The most 
common substitute is the foreseeability rule (dolus indirectus), a legal pre- 
sumption designed to prove the existence of intent. 

According to the foreseeability rule presumption, the offender is pre- 
sumed to intend the occurrence of the results if during commission of the 
conduct, with full awareness, he has foreseen the occurrence of the results 
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as highly probable.s* If the object of the results is replaced with purpose, 
this presumption applies also to specific intent.** The rationale for this 
presumption is as follows: believing that the probability of a specific fac- 
tual event following from the conduct is extremely high, committing the 
conduct expresses the offender’s desire that the factual event should occur. 

For example, A holds a loaded gun pointed at B’s head. A knows that 
if she pulls the trigger, there is a high probability that the factual event of 
B’s death will occur. A pulls the trigger. A then argues in court that she 
did not want B’s death to occur, and therefore the required component 
of intent has not been fulfilled. If the court exercises the foreseeability 
rule presumption, A is presumed to have intended the occurrence of the 
result because in A’s assessment, the result of death was highly probable 
and she acted accordingly, presumably because she wanted these results 
to occur.” This presumption is commonly used in courts to prove intent. 
Indeed, unless the defendant confesses explicitly during the interrogation 
to having had the intent, the prosecution prefers to prove the intent using 
this presumption. 

Is an al robot capable of having intent in the context of criminal law? 
Because will is a vague and general term, even in criminal law, an AI en- 
tity’s capability of having intent must be examined through the lens of the 
foreseeability rule presumption. Indeed, this is the core reason for using 
the presumption in proving human intent. Two conditions must be met 
by this rule: (1) occurrence of the results must be foreseeable with a very 
high probability, and (2) the conduct must have been committed with full 
awareness. 

As already noted, strong AI entities are capable of assessing probabili- 
ties of the occurrence of factual events and acting accordingly. For exam- 
ple, chess-playing computers can analyze the current status of the game 
based of the location of the pieces on board. These computers review all 
possible options for the next move, and for each option they review all the 
possible responses of the other player. Then, for each response they again 
review all possible responses, and so on until the final possible move that 
ends the game with a victory for one of the players. Each possible move is 
assessed for its probability, based on which, the computer decides on its 
next move.” 

If the player were human, we would say that he has an intention of 
winning the game. We would not know with certainty that he has such 
an intention, but the course of his conduct is consistent with the fore- 
seeability rule presumption. An al computer programmed to play chess 
has a goal-driven behavior of winning chess games, as do human chess 
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players. If we can say that human chess players have the intention of win- 
ning chess games, it seems that we can say the same thing about ar chess. 
Analysis of their course of conduct in the relevant situations is entirely 
consistent with the foreseeability rule presumption. 

Any entity, human or al, that examines several options for conduct 
and then makes an aware decision to commit one of them, while assess- 
ing the probability that a given factual event would result from this con- 
duct with a high degree of probability, is considered to be foreseeing the 
occurrence of that factual event. In the case of a chess game, this seems 
quite normal to us. But there is no substantial difference, in this context, 
between playing chess for the purpose of winning and committing any 
other conduct for the purpose of bringing about some result. If the result 
and the conduct together form a criminal offense, we enter the territory 
of criminal law. 

When a computer assesses the probability that a factual event (win- 
ning a chess game, for example, or killing or injuring a person) will result 
from its conduct and chooses accordingly to commit the relevant conduct 
(moving chess pieces on the board, pulling a trigger, or moving its hydrau- 
lic arm in the direction of a human body), the computer meets the required 
conditions for the foreseeability rule presumption. The computer is there- 
fore presumed to have the intention that the results should occur. This is 
exactly how the court examines the offender’s intent in most cases, when 
the offender does not confess. 

One may ask, what is considered to be a very high probability in this 
context? The answer is the same as for humans, the only difference being 
that the computer can assess the probability more accurately than a human 
can. For example, one points a loaded gun at someone’s head. A human 
shooter evaluates that the probability of the victim’s death is high as a 
result of pulling the trigger, but most humans are incapable of assessing ex- 
actly how high the probability is. If the entity is a computer, it can assesses 
the exact probability of occurrence based on the factual data to which it 
is exposed to (the distance of the victim from the gun, the direction and 
velocity of the wind, the mechanical condition of the gun, and so on). 

If probability is assessed to be high, the computer is required to act 
accordingly, and in full awareness to commit a conduct that advances the 
relevant factual event. As already noted, ar technology has the capability 
to consolidate awareness to factual data.” Commission of the conduct is 
considered as factual data, therefore ar technology can meet both condi- 
tions needed to prove the foreseeability rule presumption. This presump- 
tion is considered to be an absolute legal presumption (praesumptio juris 
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et de jure); therefore, if its conditions are met, there is no way of refuting 
its conclusion (that is, having intent). 

This analysis of the foreseeability of a1 technology relies on strong al, 
the more specific requirement being the advanced capability of assessing 
probabilities for the purpose of decision making. Because strong AT tech- 
nologies have this capability, they are therefore capable of intent within 
the context of criminal law. Thus, ar technology can meet the intent re- 
quirement in criminal law. 

Nevertheless, some of us may still have the feeling that something is 
missing before we conclude that machines are capable of intent. This may 
be correct if we consider intent in its broader sense, as used in psychol- 
ogy, philosophy, or the cognitive sciences. But here we are examining the 
criminal liability of a1 entities, and not the wider meaning of cognition. 
Therefore, the only standards of intent that are relevant to our examination 
are the standards of criminal law. Other standards are irrelevant for the 
assessment of criminal liability imposed on either humans or ai entities. 

The criminal law definition of intent (and foreseeability) is indeed 
much narrower than corresponding definitions in other spheres of knowl- 
edge. This is true for the imposition of criminal liability not only on AI 
entities, but also on humans. The definitions of intent in psychology, phi- 
losophy, or the cognitive sciences may be relevant for the endless quest 
for machina sapiens, but not for machina sapiens criminalis, which is 
based on the definitions of criminal law." This distinction is one of the 
best illustrations of the difference between machina sapiens and machina 
sapiens criminalis. 

The evidence concerning the intent of the ar entity is based on the 
ability to monitor and record all the activities of the computer. The ac- 
tivities of the computer with regard to each stage in the consolidation of 
intent or foreseeability are monitored and recorded. Assessing probabili- 
ties and making relevant decisions based on the assessment are part of the 
computer’s activities, so there is always direct evidence for proving the 
criminal intent of an AT entity if the proof is based on the foreseeability 
rule presumption. 

If intent is proven, either directly or through the foreseeability rule 
presumption, then all other forms of volition may be proven accordingly. 
Because recklessness, consisting of indifference or rashness, represents 
a lower degree of will, it may be proven either by proving recklessness 
directly or by proving intent. There are no offenses that require only reck- 
lessness and nothing else. Because criminal liability requirements func- 
tion as a matrix of minimal requirements, as noted earlier," every require- 
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ment represents the minimum condition for the imposition of criminal 
liability. The prosecution may choose between proving the given require- 
ment or any higher (but not lower) one. 

Consequently, offenses that require recklessness as their mental ele- 
ment requirement can be satisfied by proving either intent (directly or 
through the foreseeability rule presumption) or recklessness. Thus, if the 
given al technology has the foreseeability capabilities already described, it 
can meet the mental element requirements of both intent and recklessness 
offenses. But AT technology can also meet the recklessness requirement 
directly. Analogously, if the capability for intent is present, the capability 
for recklessness, which is a lower one, is also present. 

Indifference is a higher level than recklessness, and it represents a 
fully aware state of volitional neutrality toward the occurrence of the fac- 
tual event. For the indifferent person, the significance of the factual event 
occurring and not occurring is exactly the same. This has nothing to do 
with the actual probability of the occurrence of the factual event, only with 
the offender’s internal volition toward such occurrence. For example, A 
and B are playing Russian roulette.“ When it is B's turn, A is indifferent to 
the possibility of B’s death; A does not care whether B lives or dies. 

For indifference to be considered as such in the context of criminal 
law, it must be aware; that is, the offender must be aware of the various 
options and have no preference for any of them.” The decision-making 
process in strong A1 technology is based on assessing probabilities, as we 
have discussed. When the computer makes a decision to act in a certain 
way, but this decision does not take into consideration the probability that 
a given factual event can occur, the computer is indifferent to the occur- 
rence of that factual event. 

In general, complicated decision-making processes in AI technol- 
ogy are characterized by many factors to be considered. In similar situ- 
ations, humans tend to ignore a portion of the factors and not consider 
them. Computers do the same. Some are programmed to ignore certain 
factors, but strong ai technology is capable of learning to ignore factors — 
otherwise, the decision-making process would be impossible. The learn- 
ing process used by strong a1 technology, machine learning, is inductive 
learning from examples. The more examples analyzed, the more effective 
the learning.® Humans call this “experience.” 

Whenever the decision maker is an ai entity, the decision-making pro- 
cess is monitored, so there is no evidential problem in proving the indif- 
ference of the ai entity toward the occurrence of a given factual event. The 
awareness of the al entity to the possibility of the occurrence of the event 


ai criMinal liability for inTenTional offenses - 63 


is monitored, together with the factors that were taken into consideration 
in the decision-making process. This data provides direct proof of the in- 
difference of the ar entity. Indifference can also be proven through the 
foreseeability rule presumption, as noted earlier. 

Rashness is the lower level of recklessness. It consists of aware volition 
for the relevant factual event not to occur, but at the same time the com- 
mitment of aware and unreasonable conduct that causes it to occur. Al- 
though the occurrence of the factual event is undesired by the rash person, 
she conducts herself with unreasonable risk so that the event eventually 
occurs. Rashness is considered a degree of will, because if the rash of- 
fender had not wanted the event to occur at any price, she would not have 
taken unreasonable risks in her conduct. This is the major reason that 
rashness is part of the volitive aspect of mens rea. Otherwise, negative will 
alone does not justify criminal liability. 

For example, someone drives on a narrow road behind a slow truck. 
It is a two-way road divided by a solid white line. The driver is in a hurry 
and eventually decides to pass the truck, crossing the solid line. She does 
not intend to kill anyone; all she wants is to pass the truck. But a motor- 
cyclist happens to come from the opposite direction. The driver is unable 
to avoid the motorcyclist, and kills him. If the driver had wanted the mo- 
torcyclist to be dead, she would have been criminally liable for murder, 
but this was not the case. Nor would it be true to say that she was indiffer- 
ent to the death of the motorcyclist. She was rash, however. She did not 
want to hit the motorcycle, but she took unreasonable risks that resulted 
in death. She is therefore criminally liable for manslaughter. 

In the context of criminal law, rashness requires awareness. The of- 
fender is required to be aware of the possible options, to prefer not to cause 
the occurrence of the resulting event, but to commit conduct that is unrea- 
sonable if she were to avoid the occurrence of the event.™ The decision- 
making process of strong ar technology is based on assessing probabilities, 
as we have discussed. When the computer makes a decision to act in a 
certain way, but it does not weigh one of the relevant factors as sufficiently 
significant, it is considered to be rash regarding the occurrence of a given 
factual event. In our example, if the car driver were replaced by a driving 
computer, and if the computer calculated the probability of colliding with 
the motorcycle by crossing the solid line to be low, the decision to pass 
would be considered rash. 

As already noted, complicated decision-making processes in AI are 
characterized by many factors to be considered. In situations like this, 
humans sometimes miscalculate the weight of some factors. So do com- 
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puters. Humans reinforce their decisions by hopes and beliefs, but com- 
puters do not use such methods. Some computers are programmed to 
weigh certain factors in a certain way, but strong ar technology can learn 
to weigh factors correctly and accurately. Again, this learning by strong 
AT is called machine learning. 

Whenever the decision maker is an ai entity, the decision-making pro- 
cess is monitored; therefore, there is no shortage of evidence in proving 
rashness on the part of an al entity regarding the occurrence of a given fac- 
tual event. The al entity’s awareness of the possibility of the occurrence of 
the event is monitored, together with the factors that were taken into con- 
sideration in the decision-making process, and also the actual weight they 
were assigned in the given decision. This data provides direct proof of 
the rashness of the ar entity. As noted earlier, rashness can also be proven 
through the foreseeability rule presumption. 

Therefore, all components of the volitive aspect of mens rea apply to 
AT technology and can be proven in court. So the question is, who is to be 
held criminally liable for the commission of this type of offense? 


2.3. Cri Minally liable enTiTies for inTenTional ai offenses 


In general, imposition of criminal liability for mens rea offenses (“in- 
tentional offenses”) requires the fulfillment of both factual and mental 
elements of these offenses. Humans are involved in the creation of AT 
technology and entities, in their design, programming, and operation. Con- 
sequently, when an al entity meets the factual and mental elements of the 
offense, who is to be held criminally liable? The possible answers are the 
Al entity, the humans involved with the al entity, or both. Let us examine 
each of these answers. 


2.3.1.ai entity | iability 

If a human offender meets both the factual and mental element require- 
ments of an offense, criminal liability for that offense is imposed. In 
doing so, the court does not need to investigate whether the offender was 
“evil,” or whether some other attribute characterized the commission of 
the offense. The fulfillment of these requirements is the only condition 
for imposing criminal liability. Other information may affect the punish- 
ment, but not the criminal liability. We prefer our criminal legal system 
to act in this way because evil and good are matters of perspective.® Dif- 
ferent people with differing perspectives regard these terms in different 
ways.6 
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The factual and mental elements are neutral in this context, and do 
not necessarily have evil or good content.® Meeting their requirements is 
much more “technical” than the detection of evil would be. For example, 
we prohibit murder. Murder is causing the death of a human with the 
awareness and intent of causing death. If an individual factually causes 
another person’s death, the factual element requirement is fulfilled. If the 
conduct was committed with full awareness and intent, the mental ele- 
ment is also fulfilled, and the individual is criminally liable for murder, 
unless one of general defenses is applicable (for example, self-defense or 
insanity). The reason for the murder is immaterial for the imposition of 
criminal liability, and it makes no difference whether the murder was com- 
mitted out of mercy (euthanasia) or out of villainy. 

This is the way that criminal liability is imposed on human offenders. 
If we embrace this standard in relation to al entities, we can impose crimi- 
nal liability on them as well. This is the basic idea behind the criminal 
liability of ar entities, and it differs from their moral accountability, social 
responsibility, or even civil legal personhood. The narrow definition of 
criminal liability makes it possible for the AT entity to become subject to 
criminal law. Nevertheless, the reader may feel that something is still miss- 
ing in this analysis, that perhaps it falls short. We try to refute these feel- 
ings using rational arguments. 

One such feeling may be that the capacity of an al entity to follow a 
program is not sufficient to enable the system to make moral judgments 
and exercise discretion, although the program may contain a highly elabo- 
rate and complex system of rules.“ This feeling may relate eventually to 
the moral choice of the offender. The deeper argument is that no formal 
system can adequately make the moral choices that may confront an of- 
fender. Two answers apply to this argument. 

First, it is not certain that formal systems are morally blind. There are 
many types of morality and moral values.” Teleological morality, such as 
utilitarianism, deals with the utility values of conduct. These values may 
be measured, compared, and decided upon based on quantitative compari- 
sons. For example, an AT robot controls a heavy wagon that malfunctions, 
leaving two possible paths for the wagon to follow. The robot calculates 
the probabilities, and determines that following one path will result in the 
death of one person, and following the other path will result in the death 
of fifty people. Teleological morality would direct any human to choose 
the first path, and the person who followed that path would be considered 
a moral person. The robot can be directed to act in accordance with this 
morality. The robot’s morality is dictated by its program, which makes it 


When robots Kill - 66 


evaluate the consequences of each possible path. Do humans not act the 
same way? 

Second, even if the first answer is not convincing, and formal sys- 
tems are considered incapable of morality of any kind, criminal liability 
is still neither dependent on any morality nor fed by it. Morality is not 
even a condition for the imposition of criminal liability. Criminal courts 
do not asses the morality of human offenders when imposing criminal li- 
ability. The offender may be highly moral in the court’s view, and still be 
convicted, as in the case of euthanasia, for example. Alternatively, the of- 
fender may be very immoral in the eyes of the court, and still be acquitted, 
as in the case of adultery. Given that no morality of any type is required 
for the imposition of criminal liability on human offenders, it should not 
be a consideration when aI entities are involved, either. 

Some may feel that ar entities are not humans, and that criminal li- 
ability is intended for humans only because it involves constitutional 
human rights that only humans may have.” In this context, it is immate- 
rial whether the constitutional rights refer to substantial rights or to pro- 
cedural rights. The answer to this argument is that perhaps criminal law 
was originally designed for humans, but since the seventeenth century, it 
has not been applied to humans exclusively, as we have discussed.” Cor- 
porations, which are nonhuman entities, are also subject to criminal law, 
and not only to criminal law. For the past four centuries, criminal liability 
and punishments have been imposed on corporations, albeit with some 
occasional adjustments. 

Others may argue that although corporations have been recognized as 
subject to criminal law, the personhood of AT entities should not be recog- 
nized because humans have no interest in recognizing it.” This argument 
cannot be considered applicable in an analytical legal discussion on the 
criminal liability of ar entities. There are many cases in everyday life in 
which the imposition of criminal liability does not benefit human soci- 
ety, and still criminal liability is imposed. A famous example comes from 
Immanuel Kant, who is interpreted as believing that if the last person on 
Earth is an offender, he should be punished even if it leads to the extinc- 
tion of the human race.” Human benefit has not been recognized as a valid 
component of criminal liability. 

Finally, some may feel that the concept of awareness presented here 
is too shallow for at entities to be called into account, blamed, or faulted 
for the factual harm they may cause.” This feeling is based on confusion: 
the concept of awareness and consciousness in psychology, philosophy, 
theology, and cognitive sciences is confused with the concept of awareness 
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in criminal law. In other spheres of knowledge, we lack a clear notion of 
what awareness is, making it impossible to offer coherent answers to the 
question of ar capability of awareness. Whenever such answers are given, 
they are usually based on intuition rather than on science.”8 

Criminal law, however, must be accurate. Based on definitions pro- 
vided by criminal law, people may be jailed for life, and they may lose 
their property or even their lives. Therefore, criminal law definitions must 
be proven beyond any reasonable doubt. Imposition of criminal liabil- 
ity for mens rea offenses is based on awareness as the major and domi- 
nant component of the mental element requirement. The fact that the term 
awareness in psychology, philosophy, theology, and cognitive sciences has 
not been developed adequately in order to create an accurate definition 
does not exempt criminal law from developing such a definition of its own 
for the purpose of imposing criminal liability. 

The criminal law definition of awareness, like all other legal defini- 
tions, may differ significantly from its common meaning or from its mean- 
ing in psychology, philosophy, theology, and cognitive sciences. Criminal 
law definitions are designed and adapted to meet the needs of criminal law 
and nothing more. These definitions represent the requirements necessary 
for the imposition of criminal liability. They also represent the minimal 
conditions, both structural and substantive, based on the concept of crimi- 
nal law as a matrix of minimal requirements, as discussed earlier.” 

Consequently, the definitions of criminal law, including the definition 
of awareness, are relative and relevant only to criminal law. Because they 
are formed in conformity with the matrix of minimal requirements, they 
may be regarded as shallow. But this is true only when examined from the 
perspectives of psychology, philosophy, theology, and cognitive sciences, 
not from that of criminal law. If a significant development were to occur 
in those scientific spheres with regard to awareness, the criminal law may 
embrace newer and more comprehensive definitions of awareness in- 
spired by these scientific discoveries. For the time being, there are none. 

Criminal law definitions, including that of awareness, were originally 
designed for humans. For centuries people were indicted, convicted, and 
acquitted based on these definitions. Since the seventeenth century, these 
definitions have been adopted and adapted to the incrimination of non- 
human entities as well, namely corporations, which have also been in- 
dicted, convicted, and acquitted based on these definitions. Whenever 
the criminal law definitions were changed, they were changed for both 
humans and corporations in the same way. In the twenty-first century, the 
same definitions are required for the incrimination of AT entities. 
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Examination of these definitions reveals that they are applicable to 
Al technology, which can meet the requirements of criminal law without 
a single change to these definitions. Does that suddenly make them “shal- 
low,” when the same definitions have been used by criminal law systems 
worldwide for centuries and were considered adequate and almost un- 
assailable? If they are good enough for humans and corporations, why 
would not they be good enough for ai technology? 

If criticism claimed that criminal law definitions are too shallow and 
must therefore be radically changed for humans, corporations, and Ar tech- 
nology, the argument might be acceptable. But when these definitions are 
called shallow only in relation to a1 technology, the argument cannot be 
considered serious or applicable. Such criticism originates in the endless 
quest for machina sapiens, and it is part of that quest. But because crimi- 
nal law is a limited field, the quest for machina sapiens criminalis ended 
successfully when ar entities showed that they are capable of complying 
with the definitions of criminal law. 

This answer applies not only to the criticism regarding the awareness 
of AT entities in criminal law, but also to the intentionality of AT entities.” 
We can now summarize that the criticism of the idea of Ar entities’ crimi- 
nal liability generally relates to two points: (1) an absence of attributes that 
are not required for the imposition of criminal liability (for example, soul, 
evil, and good); and (2) the shallowness of criminal law definitions from 
the perspective of science other than criminal law (for example, psychol- 
ogy, philosophy, theology, and cognitive sciences). Both points of criticism 
can be answered methodically. 

The first criticism can be refuted by pointing out the structural aspect 
of the matrix of criminal law, and noting that none of the absent attributes 
is required for the imposition of criminal liability on humans, corpora- 
tions, and al entities alike. The second criticism is answered through the 
substantive aspect of the matrix of criminal law, where definitions of legal 
terms from outside the sphere of criminal law are entirely irrelevant for 
settling the question of criminal liability. This opens the door for the im- 
position of criminal liability on AT entities as direct offenders. 

Acceptance of the idea of the criminal liability of AT entities for mens 
rea offenses does not end in the commission of a certain offense by AI 
entities as principal offenders. The commission of mens rea offenses can 
also take place through complicity. No less than mens rea is required for 
the imposition of criminal liability on accomplices, because there is no 
complicity through negligence or strict liability.” A joint perpetrator may 
be considered as such only in relation to mens rea offenses. Other general 
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forms of complicity (for example, incitement and accessoryship) also re- 
guire at least mens rea. 

In sum, because all general forms of complicity reguire at least mens 
rea, an AI system may be considered as an accomplice only if it actually 
formed mens rea. Making it possible to impose criminal liability on AT en- 
tities as direct offenders also recognizes the possibility of ar entities being 
accomplices, joint perpetrators, inciters, accessories, and so on, as long as 
both factual and mental element reguirements are met in full. But these are 
not the only possible criminal liabilities that can be imposed on ar enti- 
ties. Two additional ways are possible: perpetration-through-another and 
probable consequence liability. 


2.3.2. h uman | iability: p erpetration-through-a nother 

Humans, corporations, and ar technology alike may be used as instru- 
ments in the commission of offenses, regardless of their legal personhood. 
For example, one person threatens another that he will be killed if he does 
not commit a certain offense. Having no choice, the threatened person 
commits the offense. Who is considered criminally liable for the com- 
mission of that offense? The threatening person? The threatened person? 
Both? In the context of the criminal liability of AT entities, this question 
arises when the al entity is used as an instrument by another entity. In the 
future, perhaps, the other way around may be relevant as well... 

For situations of this type, criminal law has created the general form 
of criminal liability called perpetration-through-another. This may be de- 
fined as aware execution of a criminal plan through instrumental use of 
another person, who participates in the commission of the offense as an 
innocent agent or semi-innocent agent. Perpetration-through-another is a 
late development of vicarious liability in a law of complicity. Vicarious 
liability has been recognized in both criminal and civil law since ancient 
times, and it is based on the ancient concept of slavery.” 

The master, who was a legal entity and possessed legal personhood, 
was liable not only for his own conduct but also for that of all his subjects 
(slaves, workers, family, and so on). When one of these subjects commit- 
ted an offense, it was considered as if the master himself had committed 
the offense, and the master was obligated to respond to the indictment 
(respondeat superior). The legal meaning of this obligation was that the 
master was criminally liable for offenses physically committed by his sub- 
jects. The rationale for this concept was that the master should enforce 
the criminal law among his subjects. If the master failed to do so, he was 
personally liable for the offenses committed by his subjects. 
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Because the master's subjects were considered his property, he was 
liable for the harms committed by them, under both criminal and civil law. 
A subject was considered an organ of the master— his long arm. The legal 
maxim that governed vicarious liability stated that whoever acts-through- 
another is considered to be acting for himself (gui facit per alium facit per 
se). The physical appearance of the commission of the offense was imma- 
terial for the imposition of criminal liability in this context. 

This legal concept was accepted in most ancient legal systems. Based 
on it, Roman law developed the function of the father of the family (pater- 
familias), who was responsible for any crime or tort committed by mem- 
bers of the family, its servants, guards, and slaves.*! Consequently, the 
father of the family was also responsible for the prevention of criminal 
offenses and civil torts among his subjects. His incentive for doing so was 
his fear of criminal or tort liability for actions committed by members of 
his household. The legal concept of vicarious liability was absorbed into 
medieval European law. 

The concept of vicarious liability was formally and explicitly accepted 
in English common law in the fourteenth century,ë based on legislation 
enacted in the thirteenth century.“ Between the fourteenth and seven- 
teenth centuries, English common law amended the concept and ruled 
that the master was liable for the servants’ offenses (under criminal law) 
and torts (under civil law) only if he explicitly ordered the servants to 
commit the offenses, explicitly empowered them to do so, or consented to 
their doing so before the commission of the offense (ex ante), or after the 
commission of the tort (ex post). 

After the end of the seventeenth century, this firm requirement was 
replaced by a much weaker one. Criminal and civil liability could be im- 
posed on the master for offenses and torts committed by the servants even 
if the orders of the master were implicit or the empowerment of the ser- 
vants was general.® This was the result of an attempt by English common 
law to deal with the many tort cases against workers at the dawn of the first 
industrial revolution in England and of the commercial developments of 
that time. The actions committed by the master’s workers were considered 
to be the actions of the master because he enjoyed their benefits. And if the 
master enjoyed the benefits of these actions, he should be legally liable for 
the harm that might be caused by them, under both criminal and civil law. 

In the nineteenth century, the requirements were further weakened, 
and it was ruled that if the worker’s actions were committed through or as 
part of the general course of business, the master was liable for them even 
if no explicit or implicit orders had been given. Consequently, the defense 
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argument of the worker having exceeded his authority (ultra vires) was 
rejected. Thus, even if the worker acted in contradiction to the specific 
order of a superior, the superior was still liable for the worker’s actions if 
they were carried out in the general course of business. This approach was 
developed in tort law, but the English courts did not restrict it to tort law, 
and applied it to criminal law as well.* 

Thus, vicarious liability was developed under specific social condi- 
tions, under which only individuals of the upper classes had the required 
competence to be considered legal entities. In Roman law, only the father 
of the family could become a prosecutor, plaintiff, or defendant. When the 
concept of social classes began to fade, in the nineteenth century, vicarious 
liability faded away with it. 

In criminal law at the beginning of the nineteenth century, the cases 
of vicarious liability fell into three main types. The first was classic com- 
plicity. If the relations between the parties were based on real cooperation, 
they were classified as joint perpetration even if they had an employer- 
employee or some other hierarchical relation. But if within the hierarchi- 
cal relations, information gaps between the parties or the use of power 
made one party lose its ability to commit an aware and willed offense, the 
act was no longer considered joint perpetration. The party that had lost the 
ability to commit an aware and willed offense was considered an innocent 
agent, functioning as a mere instrument in the hands of the other party. 

The innocent agent was not criminally liable, and the offense was con- 
sidered perpetration-through-another, where another party had full crimi- 
nal liability for the actions of the innocent agent.” This was the basis for 
the emergence of perpetration-through-another from vicarious liability, 
and it became the second type of criminal liability derived from vicarious 
liability. The third type accounted for the core of the original vicarious 
liability. In most modern legal systems, this type is embodied in specific 
offenses and not in the general formation of criminal liability. Since the 
emergence of the modern law of complicity, the original vicarious liability 
is no longer considered a legitimate form of criminal liability. 

Since the end of the nineteenth century and the beginning of the twen- 
tieth, the concept of the innocent agent has been widened to include par- 
ties with no hierarchical relations between them. Whenever a party acts 
without awareness of its actions or without will, it is considered an inno- 
cent agent. The acts of an innocent agent could be the results of another 
party’s initiative (for example, using the innocent agent through threats, 
coercion, misleading, or lies); or the results of another party’s abuse of an 
existing factual situation that eliminates the awareness or will of the in- 
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nocent agent (for example, abuse of a factual mistake, insanity, intoxica- 
tion, or infancy). 

During the twentieth century, the concept of perpetration-through- 
another has been applied also to semi-innocent agents, typically a neg- 
ligent party that is not fully aware of the factual situation although any 
other reasonable person could have been aware of it under the same cir- 
cumstances. Most modern legal systems accept the semi-innocent agent as 
part of perpetration-through-another, so that the other party is criminally 
liable for the commission of the offense, and the semi-innocent agent is 
criminally liable for negligence. 

If the legal system contains an appropriate offense of negligence (that 
is, the same factual element requirement, but a mental element of negli- 
gence instead of awareness, knowledge, or intent), the semi-innocent agent 
is criminally liable for that offense. If no such offense exists, no criminal 
liability is imposed, although the other party is criminally liable for the 
original offense. For the criminal liability of perpetration-through-another, 
the factual element may be fulfilled by the innocent agent, but the mental 
element requirement must be fulfilled actually and subjectively by the 
perpetrator-through-another, including with regard to the instrumental use 
of the innocent agent.” 

So the question arises, if an al entity is used by another (human, cor- 
poration, or another a1 entity) as an instrument for committing an offense, 
how would criminal liability be divided between them? Perpetration- 
through-another does not consider the ai entity that physically committed 
the offense as possessing any human attributes. The ar entity is considered 
an innocent agent, but we cannot ignore an al entity’s capabilities to physi- 
cally commit the offense. These capabilities are not sufficient to consider 
the al entity as the perpetrator of the offense, because the entity lacks the 
required awareness or will. 

The capabilities of the a1 entity to physically commit the offense re- 
semble the corresponding capabilities of a mentally limited person, such 
as an infant,® a person who is mentally incompetent,” or one who lacks a 
criminal state of mind.” Legally, when an offense is committed by an in- 
nocent agent (an infant,” a person who is mentally incompetent, or one 
who lacks a criminal state of mind to commit an offense”), no criminal 
liability is imposed on the physical perpetrator. In such cases, that person 
is regarded as an instrument, albeit a sophisticated one, while the party 
orchestrating the offense (the perpetrator-through-another) is the actual 
perpetrator as a principal in the first degree, and is held accountable for 
the conduct of the innocent agent. 
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The perpetrator's liability is determined on the basis of the conduct of 
the “instrument”! and of his mental state.® The derivative question re- 
garding al entities is, who is the perpetrator-through-another? The answer 
is any person who makes an instrumental use of the A1 technology for the 
commission of the offense. In most cases, this person is human, and may 
be the programmer of the AT software, its operator, or its end user. A pro- 
grammer may design the al entity in order to commit offenses through it. 
For example, a programmer designs software for a robot for use in a factory, 
and its software is programmed to torch the factory at night when no one 
is present. The robot committed the arson, but the programmer is consid- 
ered the perpetrator. 

The end user did not program the software, but uses the ai entity, in- 
cluding its software, for her own benefit, which is expressed by the com- 
mission of the offense. For example, a user purchases a household robot 
designed to execute any order given by its master. The robot identifies the 
user as the master, and the master orders the robot to assault any intruder 
to the house. The robot executes the order exactly as instructed. This situ- 
ation is not different from that of a person who orders her dog to attack 
any trespasser. The robot commits the assault, but the user is considered 
the perpetrator. 

In both scenarios, the actual offense was physically committed by an 
Al entity. Because the programmer and the user did not perform any action 
that conforms to the definition of a specific offense, they do not meet the 
factual element requirement of the offense. The liability in perpetration- 
through-another considers the physical actions committed by the AT entity 
as if they had been committed by the programmer, the user, or any other 
person instrumentally using the ar technology. The legal basis for this 
criminal liability is the instrumental use of the AI entity as an innocent 
agent.” No mental attribute required for the imposition of criminal liabil- 
ity is credited to the AI entity.” 

When programmers or operators use an AI entity instrumentally, the 
commission of an offense by the AT entity is attributed to them. The mental 
element required for the offense already exists in their minds. The pro- 
grammer had criminal intent when he ordered the commission of the 
arson, and the user had criminal intent when she ordered the commission 
of the assault, even if these offenses were physically committed by a robot, 
which is an ar entity. When an end user makes instrumental use of an in- 
nocent agent to commit an offense, the end user is considered to be the 
actual perpetrator of the offense. 

Perpetration-through-another does not attribute any mental or human 
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mental capabilities to the AT entity. Therefore, there is no legal difference 
between an AT entity, a screwdriver, and an animal that are instrumentally 
used by the perpetrator. When a burglar uses a screwdriver to pry open a 
window, he does so instrumentally, and the screwdriver is not criminally 
liable. The “action” of the screwdriver is in practice the burglar’s. This is 
the same legal situation as when using an animal instrumentally. An as- 
sault committed by a dog by order of its master is considered as an assault 
committed by the master. 

This type of criminal liability may be suitable for two kinds of sce- 
narios. In the first scenario, an AT entity, even strong AI technology, is used 
to commit an offense without taking advantage of the advanced capabili- 
ties of the entity. In the second scenario, a weak version of ar technology 
is used, which lacks the advanced capabilities of modern AI entities. In 
both scenarios the ar entity is used instrumentally, but nevertheless, in 
both cases the ai entity is used because of its ability to execute an order 
to commit the offense. A screwdriver cannot execute such an order, but a 
dog can. A dog cannot execute a complicated order, but an ai entity can. 

Liability of perpetration-through-another is not applicable when an 
Al entity makes the decision to commit an offense based on its own ac- 
cumulated experience or knowledge, or based on advanced calculations 
of probabilities. This liability is not suitable when the software of the AT 
entity was not designed to commit a specific offense, which was commit- 
ted by the al entity nevertheless. This type of liability does not apply either 
when the A1 entity functions not as an innocent, but as a semi-innocent 
agent. Semi-innocent agents lack the mens rea component but have a 
lower mental element component, such as negligence or strict liability, as 
we will discuss later." 

Liability of perpetration-through-another may still be applicable, how- 
ever, when a programmer or user makes instrumental use of an A1 entity, 
but without using its advanced capabilities. The legal result of applying 
this liability is that the programmer and the user are criminally liable for 
the offense committed, whereas the A1 entity has no criminal liability of 
any type.’” This is not significantly different from treating AT personhood 
as mere property, albeit with sophisticated skills and capabilities.zo3 If the 
Al entity is considered a semi-innocent agent, that is, it fulfills the negli- 
gence or strict liability requirements, it is criminally liable for offenses of 
negligence or strict liability if there are recognized by the criminal law. 
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2.3.3. Joint human and ai entity | iability: 
probable c onseguence | iability 

The first type of criminal liability, presented earlier, treated the ai entity 
as the perpetrator of the offense.1%* The second treated the ar entity as a 
mere instrument in the hands of the legally-considered perpetrator.’™ This 
second type of liability is not the only one possible to describe the legal 
relations between humans and ar entities in the commission of the offense. 
The second type dealt with adhered ai entity. But what if the ar entity, 
which was not programmed to commit the offense, calculated its actions, 
which then turn out to be considered an offense? The question here con- 
cerns the human liability rather than that of the AT entity. 

For example, the programmer of a sophisticated ar entity designs it not 
to commit certain offenses. In the beginning, after being placed in opera- 
tion, the al system commits no offenses. In time, machine learning through 
induction widens the repertoire of the ai entity, which pursues new paths 
of activity. At some point, an offense is committed. In another, somewhat 
different example, the programmer designs the AT system to commit one 
given offense. As expected, the ar system commits the offense, but then it 
deviates from the original plan of the programmer and continues its delin- 
quent activities. The deviation may be quantitative (more offenses of the 
same type), qualitative (more offenses of a different type), or both. 

If the programmer had designed the entity from the beginning to commit 
the additional offenses, this would have been considered perpetration- 
through-another at most. But this is not what the programmer did. If the 
AI system consolidates both factual and mental elements of the additional 
offenses, it is criminally liable according to the first type of liability. But 
the question here concerns the criminal liability of the programmer. This 
is the main issue of the third type of liability, which we will discuss. 

The most appropriate criminal liability in these cases is the probable 
consequence liability. Originally, probable consequence liability in crimi- 
nal law was related to the criminal liability of parties to a criminal offense 
that has been committed in practice, but that was not part of the original 
criminal plan. For example, A and B plan to commit a bank robbery. Ac- 
cording to the plan, A’s role is to break into the safe, and B’s role is to 
threaten the guard with a loaded gun. During the robbery the guard resists, 
and B shoots him to death. The killing of the guard was not part of the 
original criminal plan. When the guard was shot, A was not there, did not 
know about it, did not agree to it, and did not commit the shooting. 

The legal question in this example concerns A’s criminal liability for 
homicide, in addition to his obvious criminal liability for robbery. A does 
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not satisfy either the factual or the mental element requirements of homi- 
cide, because he did not physically commit it, nor was he aware of it. The 
homicide was not part of the perpetrators’ joint criminal plan. The ques- 
tion may be expanded also to the inciters and accessories to robbery, if any. 
In general, the question of the probable consequence liability refers to the 
criminal liability of a person for unplanned offenses that were committed 
by another. Before applying the probable consequence liability to human-a1 
offenses, we must explore its features.1°° 

There are two extreme opposite approaches to this general ques- 
tion. The first calls for imposition of full criminal liability on all par- 
ties; the second calls for broad exemption from criminal liability for any 
party that does not meet the factual and mental element requirements 
of the unplanned offense. The first is considered problematic for over- 
criminalization, while the second is considered problematic for under- 
criminalization. Consequently, moderate approaches were developed and 
embraced by various legal systems. 

The first extreme approach does not consider at all the factual and 
mental elements of the unplanned offense. This approach originates in 
Roman civil law, which has been adapted to criminal cases by several legal 
systems. According to this approach, any involvement in the delinquent 
event is considered to include criminal liability for any further delinquent 
event derived from it (versanti in re illicita imputantur omnia quae se- 
quuntur ex delicto).‘°’ This extreme approach requires neither factual nor 
mental elements for the unplanned offense from the other parties, besides 
the party that actually committed the offense and accounts for both factual 
and mental elements. 

According to this extreme approach, the criminal liability for the un- 
planned offense is an automatic derivative. The basic rationale of this ap- 
proach is deterrence of potential offenders from participating in future 
criminal enterprises by widening the criminal liability to include not only 
the planned offenses, but the unplanned ones as well. The potential party 
must realize that his personal criminal liability may not be restricted to 
specific types of offenses, and that he may be criminally liable for all ex- 
pected and unexpected developments that are derived directly or indi- 
rectly from his conduct. Potential parties are expected to be deterred and 
avoid involvement in delinquent acts. 

This approach does not distinguish between various forms of involve- 
ment in the delinquent event. The criminal liability for the unplanned of- 
fense is imposed regardless of the role of the offender in the commission 
of the planned offense as perpetrator, inciter, or accessory. The criminal 
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liability imposed for the unplanned offense does not depend on meeting 
factual and mental element requirements by the parties. If the criminal lia- 
bility for the unplanned offense is imposed on all parties of the original en- 
terprise, including those who could have no control over the commission 
of the unplanned offense, the deterrent value of this approach is extreme. 

Prospectively, this approach educates individuals to keep away from 
involvement in delinquent events, regardless of the specific role they may 
potentially play in the commission of the offense. Any deviation from the 
criminal plan, even if not under the direct control of the party, is basis for 
criminal liability for all individuals involved, as if it had been fully per- 
petrated by all parties. The effect of this extreme approach can be broad 
and encompassing. Parties to another (third) offense, different from the 
unplanned offense, who were not direct parties to the unplanned offense, 
may be criminally liable for the unplanned offense as well, if there is the 
slightest connection between the offenses. The criminal liability for the 
unplanned offense is uniform for all parties and requires no factual and 
mental elements. Most Western legal systems consider such a deterrent 
approach too extreme, and have rejected it.1% 

The second extreme approach is the exact opposite of the first, and 
focuses on the factual and mental elements of the unplanned offense. Ac- 
cording to this approach, to impose criminal liability for the unplanned 
offense, both factual and mental element requirements must be met by 
each party. Only if both requirements of the unplanned offense are met by 
a given party is it legitimate to impose criminal liability on it. Naturally, 
because the unplanned offense was not planned, it is most unlikely that 
any of the parties would be criminally liable for that offense, except for 
the party that actually committed it. This extreme approach ignores the 
social endangerment inherent in the criminal enterprise, which includes 
not only planned offenses, but unplanned ones as well. 

In light of this extreme approach, offenders have no incentive to re- 
strict their involvement in the delinquent event. Prospectively, any party 
that wishes to escape criminal liability for the probable consequences of 
the criminal plan needs only to avoid participation in the factual aspect of 
any further offense. Such offenders would tend to involve more parties in 
the commission of the offense in order to increase the chance for commit- 
ting further offenses, and therefore most modern legal systems prefer not 
to adopt this extreme approach. 

Several moderate approaches have been developed to meet the dif- 
ficulties raised by these two extreme approaches. The core of the mod- 
erate approaches lies in the creation of probable consequence liability, 
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that is, criminal liability for the unplanned offense, whose commission is 
the probable conseguence of the planned original offense. Probable con- 
seguence refers both to mental probability from the point of view of the 
party participating in the offense, and to the factual conseguence resulting 
from the planned offense. Thus, probable conseguence liability generally 
reguires two principal conditions to impose criminal liability for the un- 
planned offense: 


1. a factual condition—the unplanned offense should be the con- 
sequence of the planned offense; 

2. a mental condition—the unplanned offense should be probable 
(foreseeable by the relevant party) as a consequence of the com- 
mission of the planned offense. 


The factual condition (“consequence”) requires the incidental oc- 
currence of the unplanned offense in relation to the planned one. There 
should be a factual causal connection between the planned and the un- 
planned offense. For example, A and B conspire to rob a bank, and ex- 
ecute their plan. During the robbery, B shoots the guard to death, an act 
that is incidental to the robbery and to B’s role in it. Had it not been for 
the committed robbery, no homicide would have occurred. Therefore, the 
homicide is the factual consequence of the robbery, and it was committed 
incidentally to the robbery. 

An incidental offense is one that was not part of the criminal plan, and 
the parties did not conspire to commit it. If the offense is part of the crimi- 
nal plan, the probable consequence liability is irrelevant, and the general 
rules of complicity apply to the parties to the offense. Unplanned offenses 
fall short of this requirement and create an under-criminalization problem. 
The probable consequence liability is an attempt to address this difficulty 
by expanding the criminal liability for unplanned offenses despite the fact 
that they are unplanned. 

The unplanned offense may be a different offense from the planned 
one, but not necessarily. The offense may also be an additional, identical 
offense. For example, A and B conspire to rob a bank by breaking into 
one of its safes. A is intended to break into the safe and B to watch the 
guard. They execute their plan, but in addition, B shoots and kills the 
guard, and A breaks into yet another safe. The unplanned homicide is 
a different offense from the planned robbery. The unplanned robbery is 
identical to the planned robbery. Both unplanned offenses are inciden- 
tal consequences of the planned offense, although one is different from 
the planned offense and the other is identical to it. The planned offense 
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serves as the causal background for both unplanned offenses, as they in- 
cidentally derive from it. OS 

The mental condition (“probable”) reguires that the occurrence of the 
unplanned offense be probable in the eyes of the relevant party, meaning 
that it could have been foreseen and reasonably predicted. Some legal sys- 
tems prefer to examine the actual and subjective foreseeability (the party 
has actually and subjectively foreseen the occurrence of the unplanned 
offense), whereas others prefer to evaluate the ability to foresee through 
an objective standard of reasonability (the party has not actually foreseen 
the occurrence of the unplanned offense, but any reasonable person in his 
state could have). Actual foreseeability parallels the subjective mens rea, 
whereas objective foreseeability parallels objective negligence. 

For example, A and B conspire to rob a bank. A is intended to break 
into the safe and B to watch the guard. They execute the plan, and B shoots 
and kills the guard while A breaks into the safe. In some legal systems, 
A is criminally liable for the killing only if A had actually foreseen the 
homicide, and in others, if a reasonable person could have foreseen the 
forthcoming homicide in these circumstances. Consequently, if the rel- 
evant accomplice did not actually foresee the unplanned offense, or any 
reasonable person in the same condition could not have foreseen it, he or 
she is not criminally liable for the unplanned offense. 

This type of approach is considered moderate because it provides an- 
swers to the social endangerment problem, and at the same time it has a 
positive relation with the factual and mental elements of criminal liabil- 
ity. The factual and mental conditions are the starting terms and minimal 
requirements for the imposition of criminal liability for the unplanned 
offense. 

Legal systems differ on the legal consequences of probable conse- 
quence liability. The main factor in these differences is the mental condi- 
tion. Some legal systems require negligence, but others require mens rea, 
and the consequences may be both legally and socially different. Moder- 
ate approaches that are close to the extreme approach, which holds that 
all accomplices are criminally liable for the unplanned offense (versanti 
in re illicita imputantur omnia quae sequuntur ex delicto), impose full 
criminal liability for the unplanned offense if both factual and mental 
conditions are met. According to these approaches, the party is criminally 
liable for unplanned mens rea offenses even if he or she may have been 
merely negligent. 

More lenient moderate approaches do not impose full criminal liabil- 
ity on all the parties for the unplanned offense. These approaches can 
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show more leniency with respect to the mental element, and match the 
actual mental element of the party to the type of offense. Thus, the negli- 
gent party in the unplanned offense is criminally liable for a negligence 
offense, whereas the party who is aware is criminally liable for a mens rea 
offense. For example, A, B, and C plan to commit robbery. The robbery 
is executed, and C shoots and kills the guard. A foresaw this, but B did not, 
although a reasonable person would have foreseen this outcome under the 
circumstances. 

All three are criminally liable for robbery as joint perpetrators. C is 
criminally liable for murder, which is a mens rea offense. A, who acted 
under mens rea, is criminally liable for manslaughter or murder, both 
mens rea offenses. But B was negligent regarding the homicide, and is 
therefore criminally liable for negligent homicide. Negligent offenders are 
criminally liable for no more than negligence offenses, whereas other of- 
fenders, who meet mens rea requirements, are criminally liable for mens 
rea offenses. 

American criminal law imposes full criminal liability for unplanned 
offense equally on all parties of the planned offense" as long as the un- 
planned offense is the probable consequence of the planned one. Ap- 
propriate legislation has been enacted to accept the probable consequence 
liability, and has been considered constitutionally valid.’ Moreover, in 
the context of homicide, American law incriminates incidental unplanned 
homicide committed in the course of the commission of another planned 
offense as murder, even if the mental element of the parties was not ad- 
equate for murder. 

By comparison, European-Continental legal systems and English 
common law impose criminal liability for the unplanned offense equally 
on all parties of the planned offense." The English'® and European- 
Continental moderate approaches are closer to the first extreme approach.” 

For the applicability of the probable consequence liability to human- 
AI offenses, we must distinguish between two types of cases, as shown in 
our opening examples of computer programmers at the beginning of this 
section. The first type refers to cases in which the programmer designed 
the ar system to commit a certain offense, but the system exceeded the pro- 
grammer’s plan quantitatively (and committed more offenses of the same 
type), qualitatively (committed more offenses of different types), or in both 
respects. The second type refers to cases in which the programmer did not 
design the AT system to commit any offense, but the system committed an 
offense nevertheless. 

In the first type of cases, criminal liability is divided into the liabil- 
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ity for the planned offense and for the unplanned one. If the program- 
mer designed the system to commit a certain offense, this is perpetration- 
through-another of that offense, at most. The programmer instructed the 
system what it should do, therefore he used the system instrumentally for 
the commission of the offense. In this case, the programmer is alone re- 
sponsible for the offense, as already discussed." For this particular crimi- 
nal liability, there is no difference between an al system, some other com- 
puterized system, a screwdriver, and any innocent human agent. 

In the case of the exceeded offenses, a different approach is required. 
If the ar system is strong, and is capable of computing the commission of 
an additional offense, the a1 system is considered criminally liable for that 
offense according to the standard rules of criminal liability, as described 
earlier.° This completes the criminal liability of the a1 system, and the 
criminal liability of the programmer is determined according to the prob- 
able consequence liability already described. Thus, if from the program- 
mer's point of view the additional offense was a probable consequence of 
the planned offense, then criminal liability is imposed on the program- 
mer for the unplanned offense in addition to the criminal liability for the 
planned offense. 

If the AT system is not considered strong, and is not capable of com- 
puting the commission of the additional offense, the a1 system cannot be 
considered criminally liable for the additional offense. Under these condi- 
tions, the ar system is considered as innocent agent. The criminal liability 
for the additional offense remains the programmer’s alone, on the same 
basis of probable consequence liability already described. Thus, if the ad- 
ditional offense was a probable consequence of the planned offense from 
the programmer’s point of view, the programmer is criminally liable for 
the unplanned offense in addition to his criminal liability for the planned 
offense. 

In the second type of cases, the programmer has no intention to commit 
any offense. From the programmer’s point of view, the occurrence of the 
offense is not more than an unwilled accident. Because the programmer’s 
initiative was not criminal, the probable consequence liability is inappro- 
priate. The presence of a planned offense is crucial for the imposition of 
probable consequence liability, as discussed, because the probable con- 
sequence liability is intended to address unplanned developments of a 
planned delinquent event and serve as a deterrent against participation in 
delinquent activities. 

When the programmer’s starting point is not delinquent, and from his 
point of view the occurrence of the offense is accidental, then deterrence 
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is inappropriate and irrelevant. Applying such a mechanism to deal with 
mistakes and accidents behind which there is no criminal intent would be 
disproportional. Thus, if the al system, which actually committed the of- 
fense, is considered strong and capable of consolidating the requirements 
of the offense, it may be criminally liable for that offense as a direct perpe- 
trator. If not, the ar system is not criminally liable for the offense. 

The programmer is at most negligent. In this type of case, the program- 
mer's criminal liability does not depend on the criminal liability of the AT 
system. Regardless of whether the a1 system is criminally liable, the crimi- 
nal liability of the programmer for the unplanned offense is examined 
separately. Because the programmer did not intend any offense to occur, 
the mental element of mens rea does not apply to him, and his criminal li- 
ability must be examined by the standards of negligence, so that he would 
be criminally liable for negligence offenses at most. The standard of neg- 
ligence is discussed in the next chapter, where we discuss the criminal 
liability of a1 entities for negligence offenses. 


closing The opening exaMople: inTenTional Killin g r oboT 


We opened this chapter with the example ofa killing robot. We can now go 
back and analyze the example based on the insights gained from this chap- 
ter. There are two main actors whose criminal liability must be evaluated 
in this homicide: the robot and its programmer. To examine the robot’s 
criminal liability, we must first assess its technological capabilities. If the 
Al robot has the technological capabilities of meeting the requirements of 
mens rea offenses, it becomes a machina sapiens criminalis in this con- 
text. If such capabilities are present, the robot’s criminal liability in this 
case should be examined as if it were human. 

Homicide (mens rea) offenses can be murder or manslaughter. Their 
factual element requirement is identical, and it includes causing the death 
of a human. Because the robot has physically caused the worker’s death, 
the factual element requirement appears to be met. The mental element re- 
quirement of murder requires intent (awareness and intent), whereas that 
of manslaughter requires recklessness (awareness and recklessness). At 
this point, careful analysis is needed. If, after analyzing the robot’s record 
of computations, one of these requirements is met, it forms the criminal 
liability of the robot for the given homicide offense. If not, negligence may 
be examined, as discussed in the next chapter, and the robot's criminal li- 
ability is for negligent homicide. But if the robot lacks the technological 
capabilities to meet the requirements of either mens rea or negligence of- 
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fenses, it does not form a machina sapiens criminalis in this context, and 
it is not criminally liable for the homicide. The only criminal liability for 
the homicide would be the programmer's and that of other related parties 
(users, the factory, and so on). Here, programmer refers not only to the 
specific person or persons who actually programmed the system, but to 
all related persons, including the company that manufactured the robot. 

The programmer's criminal liability is determined based on his role 
in bringing about the homicide. If the programmer instrumentally used 
the robot to kill the worker by designing it to operate in this way, he is 
a perpetrator-through-another of the homicide. This is also the case if 
the robot has no advanced Ar capabilities for meeting the requirements 
of the offense. In these cases, the programmer and the related parties are 
alone criminally liable for the homicide, and their exact criminal liabil- 
ity for the homicide, whether murder or manslaughter, is determined by 
the manner in which they met the mental element requirements of these 
offenses. 

If the programmer designed the robot to commit an offense (any of- 
fense) other than the homicide of the worker, and the ai system of the 
robot continued the delinquent activity through an unplanned offense (the 
homicide of the worker), then the programmer’s criminal liability for the 
homicide is determined based on probable consequence liability. The ro- 
bot’s criminal liability in this case is examined according to its capability 
to meet the requirements of the given offense. If the programmer did not 
design the robot to commit any offense, but the unplanned homicide oc- 
curred, then the robot’s criminal liability is examined based on its capabil- 
ity to meet the requirements for homicide, and the programmer’s criminal 
liability is examined according to the standards of negligence, as we will 
discuss in the next chapter. 


ai criminal Iia Bili Ty for negligence offen SeS 


One of the common applications of al expert systems is medical, generally 
used for more accurate diagnosis. Patient symptoms are entered into the 
system using visual scanners or other means to capture verbal data, after 
which the expert system analyzes the factual data and suggests a diagno- 
sis to the medical staff. Based on machine learning (inductive analysis of 
cases and generalization), the expert system gains its “experience” and 
learns how to distinguish among various types of symptoms, and also to 
ignore unimportant symptoms. The longer the system is active, the more 
accurate is its diagnosis. 

This method means, among other things, that identical cases brought 
to the system for diagnosis at different times might be diagnosed differ- 
ently in light of the experience gained by the system between diagnoses. 
The same is true of human physicians. The case of one-year-old baby was 
brought to an expert Ar system for diagnosis.” The baby suffered from high 
fever (104 degrees F), dehydration, and general weakness. The system di- 
agnosed influenza, and recommended using analgesics and an infusion of 
fluids. The medical staff acted accordingly, but after five hours the baby 
died. The autopsy revealed that the baby died of a severe bacterial infec- 
tion; if treated with the appropriate antibiotics, the baby could have sur- 
vived. The legal question is, who is to be held criminally liable for the 
baby’s death? 

Negligence offenses are in common use in most countries worldwide 
in a context of professional malpractice. To impose criminal liability in 
these offenses, both factual and mental element requirements must be met 
by the apparent offender. Consequently, the question of the criminal liabil- 
ity of al entities for negligence offenses is determined by whether they are 
capable of meeting these requirements accumulatively. But even if al enti- 
ties have such capability, their liability does not function as an exemption 
for humans from their own criminal liability, if humans were involved in 
the commission of the offense. 
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To explore these issues, we examine the capability of Ar entities to fulfill 
both the factual and mental element requirements of negligence offenses, 
as well as the criminal liability of humans for ar negligence offenses. 


3.1. The facTual eleMenT r eQuire MenT 


The factual element requirement structure (actus reus) is identical for all 
types of offenses: intentional, negligence, and strict liability, as noted ear- 
lier.* This structure contains one mandatory component (conduct) for all 
offenses, and two optional components (circumstances and results) for 
some offenses. The capability of machines to meet the factual element re- 
quirement considering this structure has already been discussed,‘ and the 
conclusions of the discussion are same within the context of negligence 
offenses. 


3.2. The MenTal eleMenT r eQuire MenT 


Imposition of criminal liability for negligence offenses requires meeting 
both factual and mental element requirements. The mental element re- 
quirement of negligence offenses is negligence. If a1 technology is capa- 
ble of meeting the negligence requirement, it is possible and feasible to 
impose on such technology criminal liability for negligence offenses. First, 
we explore the structure of the negligence requirement, and then the ful- 
fillment of the requirement by a1 technology. 


3.2.1. Structure of negligence r equirement 

Negligence has been recognized as a behavioral standard since ancient 
times. It was already mentioned in the Laws of Eshnunna in the twenti- 
eth century sc, in Roman law,’ in Canonic law, and in the early English 
common law.’ But in these legal systems, negligence was treated as a be- 
havioral standard rather than as a type of mental element in criminal law. 
The behavioral standard included dangerous behavior that fails to take 
into account all relevant considerations related to the individual’s given 
action. Only since the seventeenth century has negligence been consid- 
ered a type of mental element in criminal law. 

In 1664, the English court ruled that negligence was not adequate to 
convict a defendant for manslaughter, and that at least recklessness was 
required.® This ruling created negligence as a type of mental element in 
criminal law. During the nineteenth century, negligence was treated as an 
exception for the general requirement of mens rea." Accordingly, it was 
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required explicitly and construed strictly. The offense needed to require 
negligence explicitly, in its definition, if it were to be adequate for impos- 
ing criminal liability. In the nineteenth century, negligence offenses were 
still quite rare. 

As transportation developed, especially with the rise of automobiles, 
negligence started to be used more frequently in criminal law. Deaths 
caused by traffic accidents became more common, and manslaughter was 
not appropriate for these cases. A lower level of homicide was required, 
and the solution was found in negligent homicide.’° But when negligence 
came into common use, it caused confusion. Negligence was interpreted 
as requiring unreasonable conduct, and this resulted in its being confused 
with recklessness of the lower level (rashness), which required taking 
unreasonable risk. That confusion produced such unnecessary terms as 
“gross negligence” and “wicked negligence.” 

Many misleading rulings were issued on this basis in English law,” 
until the House of Lords clarified the distinction in 2003. American law 
developed negligence as a mental element in criminal law, parallel to the 
English common law and inspired by it.* Negligence was accepted as an 
exception to mens rea during the nineteenth century, but more accurately 
than in English law.” The main distinction between recklessness and neg- 
ligence involves the cognitive aspect of recklessness. Whereas reckless- 
ness requires the cognitive aspect of awareness, as part of the mens rea 
requirement, negligence does not." 

Both reckless and negligent offenders are required to take unreason- 
able risks. However, the reckless offender is required to be aware of the 
factual element components, but the negligent offender is not required 
to be so aware.” Negligence functions as an omission of awareness, and 
it creates a social standard of conduct. An individual is required to take 
only reasonable risks, measured objectively through the perspective of an 
abstract reasonable person. The reasonable person is aware of his or her 
factual behavior, and takes only reasonable risks.’° Naturally, it is the court 
that determines how reasonable a risk is, and it does so retrospectively in 
relation to a particular case. 

The modern development of negligence in American law is reflected 
in the American Law Institute’s Model Penal Code, inspired by modern 
European-Continental understandings of negligence. Accordingly, negli- 
gence is a type of mental element in criminal law, and as such it relates to 
the factual element components. Negligence requires lack of awareness of 
the factual element components, whereas a reasonable person could and 
should have been aware of them, and requires taking unreasonable risk 
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regarding the results of the offense. This development has been embraced 
by American courts.” 

How, then, can negligence function as a mental element in criminal 
law if the offender is not even required to be aware of his or her conduct? 
Some scholars have asked to exclude negligence from criminal law and 
leave it for tort law or other civil proceedings.” But the justification of neg- 
ligence as mental element focuses on its function as an omission of aware- 
ness. Exactly the same way that act and omission are considered identical 
for the imposition of criminal liability, as discussed earlier in relation to 
the factual element,22 so can both awareness and omission of awareness 
be considered as the basis for meeting the mental element requirement. 

Following the analogy with the factual element, negligence does not 
parallel inaction, but parallels omission. If a person was simply not aware 
of the factual element components, and nothing more, she is not consid- 
ered negligent, but innocent. Omitting to be aware means that the person 
was not aware although a reasonable person could and should have been; 
that is, the individual is considered to not be using her existing capabili- 
ties of forming awareness. If the individual was not aware but was capable 
of being aware, she also had the duty to be aware (non scire quod scire 
debemus et possumus culpa est). 

Negligence does not incriminate individuals who are incapable of 
forming awareness, but only those who failed to use their existing capa- 
bilities to do so. Negligence does not incriminate the blind person for not 
seeing, but only people who have the ability to see but failed to use this 
ability. Wrong decisions are a common part of daily human life, so neg- 
ligence does not apply to situations of this type. Taking risks is also part 
of human life, and therefore negligence is not intended to inhibit it, and 
indeed society encourages taking risks in many situations. Negligence ap- 
plies to taking unreasonable risks. 

If people were to take no risks at all, then human development would 
stop in its tracks. If we had taken no risks, we would have been staring at 
the burning branch after it was struck by lightning, afraid to reach out, grab 
it, and use it for our needs. Society constantly pushes people to take risks, 
but reasonable ones. The question is, how can we identify the unreasonable 
risks and distinguish them from the reasonable ones, which are legitimate? 

For example, scientists propose to develop an advanced device that 
would significantly ease our daily lives. It is comfortable, fast, elegant, 
and accessible, but using it may cause the deaths of about 30,000 people 
each year in the United States alone. Would developing such a device be 
considered a reasonable risk? 
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Thirty-thousand victims a year appears to be an enormous number, 
which makes using the device completely unreasonable. But using the 
device in question, which we normally call a “car,”2 is not considered 
unreasonable in most countries in the world today, although in the late 
nineteenth century it was. The same is true for trains, airplanes, ships, and 
many of our common means of transport and other everyday tools. The 
degree to which a risk is reasonable is relative to its nature, and is rela- 
tively determined with respect to time, place, society, culture, and other 
circumstances. Different courts in different countries find different types 
of people and behaviors to be reasonable within the context of negligence. 

Whether a person was reasonable must be determined not only in the 
abstract, but also with reference to the relevant circumstances of the spe- 
cific offender. For example, it is not appropriate to compare medical deci- 
sions of a physician with the behavior of an abstract reasonable person; 
such decisions should be compared with those of a reasonable physician 
of the same level of expertise and experience, acting in the same or similar 
clinical circumstances (emergency or other), having the same resources at 
his or her disposal, and so on. Such an approach would make the stan- 
dard of reasonability more focused, but also subjective rather than purely 
objective. A detailed discussion of this process in relation to al systems 
follows.” 

Most negligence offenses are result offenses because society prefers to 
use negligence to protect against factual harms resulting from unreason- 
able risk taking. But negligence may be required for conduct offenses as 
well. The structure of negligence is described schematically in table 3.1. 

The general structure of negligence has no volitive aspect, only a 
cognitive aspect. Because volition is supported by cognition, and negli- 
gence does not require awareness, negligence cannot require volitive com- 
ponents. The cognitive aspect of negligence consists of lack of awareness 
of all factual element components. The requirements of negligence in re- 
lation to conduct and circumstances are identical. Both require lack of 
awareness of the component (conduct/circumstances) despite the capabil- 
ity to form such awareness, when a reasonable person could and should 
have been aware of that component. 

The reasonability in the components of negligence is examined with 
reference to the capability and duty of the offender to form awareness, 
although in practice, no awareness has been formed by him or her. With 
respect to the results, the negligence requirement requires lack of aware- 
ness of the possibility of the results occurring, despite the capability to 
form such awareness, when a reasonable person could and should have 
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Table 3.1 -+ General Structure of Negligence 


fac TUal elemen Tcomponen T negligence componen T 


Lack of awareness of conduct 
Conduct despite the capability to form such 
awareness, when a reasonable per- 
son could and should have been 
aware of that conduct. 


Lack of awareness of circumstances 
Circumstances despite the capability to form such 
awareness, when a reasonable per- 
son could and should have been 
aware of the circumstances. 


Lack of awareness of the possibil- 
Results ity of the occurence of the results 
despite the capability to form such 
awareness, when a reasonable per- 
son could and should have been 
aware of that possibility as an unrea- 
sonable risk. 


been aware of that possibility as an unreasonable risk. For this component, 
the reasonability focuses on identifying the occurrence of the result as a 
possibility involving unreasonable risk. In other words, under the given 
circumstances, the offender took unreasonable risks in the event at hand. 

The modern structure of negligence is consistent with the concept of 
criminal law as a matrix of minimal requirements, described earlier.” It 
contains both internal and external aspects. Inwardly, negligence is the 
minimal mental element requirement for each of the factual element com- 
ponents. Consequently, if negligence is proven in relation to circumstances 
and results, but awareness is proven in relation to the conduct, this satis- 
fies the requirement of negligence. This means that for each of the factual 
element components, at least negligence is required, but not exclusively. 
Outwardly, the mental element requirement of negligence offenses is sat- 
isfied through at least negligence, but not exclusively. This means that 
criminal liability for negligence offenses may be imposed by proving mens 
rea as well as negligence. 

Because negligence is still considered an exception to the general re- 
quirement of mens rea, negligence has been required as the mental ele- 
ment of relatively light offenses. In some legal systems, negligence has 
even been restricted ex ante to light offenses.”® This general structure of 
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negligence is a template that includes terms from the mental terminology 
(for example, reasonability). To explore whether a1 entities are capable of 
fulfilling the negligence requirement of given offenses, we must examine 
these terms. 


3.2.2. f ulfillment of the negligence r equirement: 
is objectivity Subjective? 

Sonja: What prevents you from murdering somebody? 
Boris: Murder’s immoral. 
Sonja: Immorality is subjective. 
Boris: Yes, but subjectivity is objective. 

— Diane Keaton and Woody Allen as Sonja and Boris 

Grushenko in Love and Death, United Artists, 1975 


The core of the negligence template in relation to the factual element com- 
ponents is expressed by lack of awareness of the factual component de- 
spite the capability to form such awareness, when a reasonable person 
could and should have been aware of that component. Lack of awareness 
is naturally the opposite of awareness, which is required by mens rea, as 
discussed earlier.” Consequently, for a person to be considered aware of 
certain factual data, two accumulative conditions are required: (1) absorb- 
ing the factual data through the senses; and (2) creating a relevant general 
image with regard to this data in the brain. If either of these conditions is 
missing, the person is not considered to be aware. 

Lack of awareness may be achieved by the absence of at least one of 
these two conditions. If the factual data has not been absorbed by the 
senses, or if it was absorbed but no relevant general image has been cre- 
ated, then the situation is considered one of lack of awareness to that fac- 
tual data. Awareness is a binary situation in the context of criminal law; 
therefore, no partial awareness is recognized. If portions of the awareness 
process are present but the process has not been accomplished in full, the 
person is considered to be unaware of the relevant factual data. This is true 
for both human and Ar offenders. 

For lack of awareness to be considered omission of awareness, it must 
occur despite the capability of the offender to form awareness, when a rea- 
sonable person could and should have been aware. These are two separate 
conditions: (1) possessing the cognitive capabilities needed to consolidate 
awareness; and (2) a reasonable person could and should have been aware 
of the factual data. The first condition has to do with the offender’s physi- 
cal capabilities. If the offender lacks these capabilities, regardless of the 
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offense, no criminal liability for negligence may be imposed on him, just 
as we cannot punish the blind for not seeing. 

Therefore, negligent offenders are only those who possess the capabil- 
ity of forming awareness. That is true for both human and ai offenders. 
Therefore, to impose criminal liability for a negligence offense on an AI 
entity, the system must possess the capability of forming awareness. An 
AI system that lacks such capability cannot be held liable for an offense 
of negligence, and naturally it cannot be held liable for mens rea offenses 
either. These capabilities must be proven based on the general features of 
the al system, regardless of the case at hand. 

If we have established that the offender was not aware of the relevant 
factual data, and that he has the capability of being aware, for the offense 
to become negligence we must still prove that a reasonable person could 
and should have been aware of the factual data under similar circum- 
stances. The “reasonable person” is a mental construct established for the 
purpose of comparison. Although in some other legal spheres, the reason- 
able person refers to a standard higher than that of the average person, in 
criminal law it refers to the average person.”* Different societies and cul- 
tures, at different times and in different places, assign different content 
and qualities to the reasonable person. 

The reasonable person is expected to reflect the existing situation in 
the given society, and not be used by the courts to change the current situ- 
ation. The standard relates to the cognitive processes that should have oc- 
curred. The reasonable person is assessed based on two paths of cognitive 
activity: he or she should (1) take into account all relevant considerations, 
and (2) assign the proper weight to these considerations. Therefore, ig- 
norance of relevant considerations is considered to be unreasonable, and 
after all relevant considerations have been taken into account, they must 
also be properly weighed for the action to be considered reasonable. 

In general, the relevance of the considerations and their proper weight 
must be determined by the court ex post. The complexity and variety 
of common life situations makes it impossible to characterize a general 
type of reasonable person. Such a type would be purely objective and at 
the same time unrealistic and irrelevant to too many real-life situations. 
Therefore, the objective standard for the reasonable person must be set 
with some degree of subjectivity if it is to match any particular case. The 
subjective setting is the link between the individual offender and the ob- 
jective and general standard of a reasonable person. 

Thus, the reasonable person must be not only assessed by the standards 
of a general abstract person, but adapted to the specific circumstances of 
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the individual offender. We expect different people to behave differently. 
Experienced attorneys act differently from inexperienced ones, even in 
the same situations. The same is true for pilots, drivers, physicians, police 
officers, and indeed, for all of us. Moreover, the same person acts differ- 
ently under differing circumstances, weighing considerations differently. 
A soldier under enemy attack, in an emergency, in a life-threatening situ- 
ation, acts differently than during routine training. 

Therefore, the reasonable person relevant for a given case is not a gen- 
eral standard of objectivism, but it includes subjective reflections of the 
individual offender.2? A surgeon with ten years of experience, acting in 
an emergency situation and using the limited resources available in the 
surgery, would be compared with a reasonable surgeon having the same 
attributes. The attributes relevant for that comparison, as well as their con- 
tent and effect, are determined by the court, which may resort to the expert 
opinion of professionals for this purpose. 

The reasonable person forms a sphere of reasonability that contains 
all types of reasonable behaviors in the given situation. The assumption 
is that there are several reasonable ways to behave in these situations, 
and only deviations from the sphere of reasonability can form negligence. 
When the factual data relate to the results of the conduct, the possibility 
of the occurrence of the results is considered an unreasonable risk. In this 
context, taking unreasonable risks is considered acting outside the sphere 
of reasonability with respect to the results.°° 

Reasonable and unreasonable risks are assessed in the same way as 
reasonable and unreasonable persons, as just described. For a risk to be 
considered reasonable, the individual must take into account all relevant 
considerations and assign the proper weight to them. Based on these con- 
siderations, if taking the risk is one of the available options, then the risk 
is considered reasonable; if not, it is considered unreasonable. To meet the 
negligence requirement, an AI entity must make unreasonable decisions. 
The ultimate question is whether a machine can be reasonable, or perhaps 
whether a machine can be unreasonable. 

From an analytical point of view, the reasonability of a machine is not 
different from that of a human. Both must take into account the relevant 
considerations and assign the proper weight to them. This can easily be a 
matter of calculation. The relevant considerations are not more than terms 
in the equation, and their proper weight is the combinations of these terms. 
The equation may be constant if so programmed, but machine-learning 
features can change that. Machine learning is a process of generalization 
by induction from many specific cases, as we have already discussed.’ 
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Machine learning enables the al system to change the equation from time 
to time. 

Indeed, effective machine learning should cause changes in the equa- 
tion almost every time a given case is analyzed. This is what happens to 
our image of the world as our life experience becomes richer and broadens. 
Without machine learning, the equation remains constant and the system 
becomes ineffective. Expert systems that are not capable of machine learn- 
ing are not different from human experts who insist on not updating their 
own knowledge. Machine learning is essential for the ar system to con- 
tinue developing. 

When the ai system is activated for the first time, the equation and its 
terms are programmed by human experts who determine what is a reason- 
able course of conduct in individual cases. After analyzing a few cases, 
the system begins to identify exceptions, wider and narrower definitions, 
new factors and new connections between existing ones, and so on. The 
AI system generalizes the knowledge absorbed from individual cases by 
reformulating the relevant equations. We use the term equation to describe 
the relevant algorithm, but it is not necessarily an equation in the mathe- 
matical sense. 

Reformulating equations raises the possibility that the ar entity will 
make different decisions in the future than it made in the past. This pro- 
cess of induction is at the core of machine learning, and changes in the 
equation produce each time a different sphere of right decisions. For ex- 
ample, if on its first activation, a medical expert system is given a list of 
symptoms of both common cold and influenza, it produces a diagnosis 
based on those symptoms alone. But after being exposed to more cases, the 
system learns to take into account other symptoms that may be essential 
for distinguishing between the two conditions. 

If the system is required to recommend medical treatments, these will 
differ for different diagnoses. At times, the expert system will not be “sure” 
because the symptoms can match more than one disease, and the system 
will assess the probabilities based on measurement and analysis of the 
various factors. For example, the expert system may determine that there 
is a probability of 47 percent that the patient has a common cold, and of 
53 percent that she has influenza. Processing these probabilities may be 
the cause of negligence on the part of al systems, and of mistaken conclu- 
sions, both sure and unsure. The system may be sure in the case of a mis- 
taken conclusion, and can also assess probabilities mistakenly. 

Mistakes may be caused by inauspicious changes of the equation, 
wrong factors being considered, ignorance of certain factors, or wrong 
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weights assigned to some factors. These mistakes are the by-product of 
errors in the machine-learning process, or more precisely, ex post errors, 
which are considered errors only after the decision has been made and 
based on the consequences of that decision. Humans tend to learn empiri- 
cally by trial and error. From the analytical point of view, in this context 
machine and human errors are of the same type. Understanding the error, 
its causes, and the ways to avoid it in the future are part of the learning 
process, for both humans and machines. 

The question is, what is to be considered a reasonable decision in this 
context? The principal question is whether, in light of the system’s ini- 
tial state with respect to the basic factual data and the experience it has 
acquired through machine learning, a reasonable person could be aware 
of the relevant factual data. And the derivative question is, who is this 
reasonable person: a human or a machine? If we accept the concept that 
objectivity in negligence offenses is subjective to some degree, as already 
discussed, then the reasonable person must have attributes similar to those 
of the offender. Only then can the reasonability of the offender’s decision 
be assessed without causing injustice. 

Therefore, if the offender is capable of machine learning, so should be 
the reasonable person, according to this concept. It follows that in deter- 
mining the reasonability of the ar entity’s decisions, the reasonable person 
would be a reasonable ai system of the same type. It may be problematic 
for the human programmers, operators, and users to escape criminal li- 
ability and impose it on the mistaken machine. It would be convenient for 
the medical staff to use an expert system and follow its recommendations, 
then if the system is wrong, to have criminal liability imposed on it alone. 
But the legal situation is not that simple. 

The decisions of assigning an AI system to its position, using it, follow- 
ing its recommendations, and so on are all subject to negligence offenses 
as well. The AT system is capable of meeting the negligence requirements, 
but this does not exempt other individuals involved in the situation from 
criminal liability. The very decision to use the al system is itself subject to 
criminal liability. For example, if the decision was made with full aware- 
ness of particular mistakes that the system can make, and if these mistakes 
resulted in death, the human decision may lead to a charge of murder. But 
if there was no such awareness, although a reasonable person in a similar 
situation could and should have been aware of the risks, this may lead to 
a charge of negligent homicide. 

Evaluation of the reasonable machine has to do with the machine- 
learning features of the system. The imposition of criminal liability in 
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negligence offenses is based on analysis of the machine-learning process 
that led to the mistaken decision. The records maintained by the ai system 
itself are the basis for the analysis. But the reasonability of the decision- 
making process within the framework of machine learning may be based 
on expert opinion. This is how negligence is proved in court in the case 
of human offenders. It is not uncommon to prove or to refute the human 
negligence using expert opinions. 

For example, when the medical expert system produces probabilities 
of 47 percent for common cold and 53 percent for influenza, a medical 
expert may explain to the court why these probabilities are reasonable 
or unreasonable for the case at hand, and a computer scientist may ex- 
plain to the court the process of producing these probabilities based on the 
machine’s learning process and existing database. Accordingly, the court 
must ask three questions: 


1. Was the al system unaware of the factual components? 

2. Does the a1 system have the general capability of consolidating 
awareness of the factual components? 

3. Could a reasonable person have been aware of the factual com- 
ponents? 


If the answer is affirmative to all three questions, and this is proven 
beyond any reasonable doubt, the ar system has met the requirements of 
the negligence offense. ar systems that are capable of forming awareness 
of mens rea offenses, as discussed earlier, are not limited either techno- 
logically or legally in consolidating negligence for negligence offenses, 
because negligence requires a lower level of mental element than mens 
rea does. Negligence is therefore relevant to ar technology, and it is pos- 
sible to prove it in court. But we must still answer the question of who is 
criminally liable for the commission of this type of offenses. 


d.d. CriMinally liable entities for ai negligence offenses 


In general, imposition of criminal liability for negligence offenses requires 
meeting both the factual and mental element requirements of these of- 
fenses. Humans are involved in the creation, design, programming, and 
operation of ar technology and entities. Therefore, when the factual and 
mental elements of the offense are fulfilled by an ai entity, the question 
is, who is to be criminally liable for the offenses committed? The possible 
answers may be the al entity, the related humans, or both. We will now 
discuss these answers. 
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3.3.1.ai entity | iability 

In negligence, as in mens rea offenses, when the offender meets both the 
factual and mental element requirements of an offense, criminal liability 
is imposed. As in mens rea offenses, the court is not expected to check 
whether the offender was “evil,” “immoral,” or so on, as noted earlier.53 
This is true for every type of offender: humans, corporations, and ai enti- 
ties. Therefore, the justifications for the imposition of criminal liability 
on AI entities in mens rea offenses are relevant in the case of negligence 
as well. As long as these requirements are even narrowly met, criminal li- 
ability can be imposed. 

Negligence offenses, however, differ from mens rea offenses in their 
social purpose. The pertinent question is whether their differing social 
purpose is relevant not only for humans and corporations, but also for 
Al entities. Initially, negligence offenses were not intended to deal with 
“evil” people, but with individuals who make mistakes. Therefore, the 
debate about villainy in criminal law is not relevant to negligence offenses, 
as it may be for mens rea offenses. In the context of negligence offenses, 
criminal law serves to help draw the boundaries of individual discretion. 

Any person can make a wrong judgment, most of the time without 
contradicting the norms of criminal law. For example, we may choose the 
wrong spouse, car, employer, apartment, or even faith. But when mistaken 
use of our discretion leads to someone’s death (negligent homicide), we do 
breach a norm of criminal law. 

As long as our exercise of wrong judgment does not contradict criminal 
law, society expects us to learn our lesson on our own, so that the next time 
we choose a car, house, employer, and so on, we will be more careful. This 
is how we gain life experience. In these cases, society takes the risk that we 
may not learn our lesson, but it does not intervene through criminal law. 
Yet when criminal offenses are committed, society does not take the risk 
of allowing the individual to be home-schooled, because the social harm 
is too serious to assume such risk. 

In these cases, society intervenes by means of negligence offenses. The 
purpose of these offenses is to increase the likelihood that the individual 
will learn the relevant lesson. Prospectively, it is assumed that after the 
lesson has been taught, the probability of committing the same offense 
again is much lower. In this way, society educates physicians to be more 
careful during surgery, employers to be more assiduous in protecting their 
employees’ lives and welfare, construction companies to be more scrupu- 
lous in observing safety regulations, factories to create less pollution, and 
so on. Human and corporate offenders are expected to learn their lessons 
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through the intermediary of criminal law. Should this type of education 
be relevant to ar entities as well? 

The answer is affirmative. For the educational purpose of the criminal 
negligence offense, it is true that there is not much use in the imposition 
of criminal liability unless the offender has the ability to learn. If we want 
the offender to learn a lesson after making a mistake, we must assume that 
the offender is capable of learning. If such capability is present and exer- 
cised, then criminal liability for negligence offenses is necessary. But if no 
such capability is exercised, then there is no prospective value in impos- 
ing criminal liability for negligence because the results would be the same, 
regardless of whether criminal liability is imposed. 

For al systems, which are equipped with the capabilities necessary for 
machine learning, criminal liability for negligence offenses is necessary. 
As for humans, negligence offenses can draw the boundaries of discretion 
of AT systems. Humans, corporations, and al systems alike are expected to 
learn from their mistakes and improve their decisions prospectively. When 
the mistakes fall under its purview, criminal law intervenes in shaping 
the decision maker’s discretion. For A1 systems, criminal liability for neg- 
ligence offenses is a chance to reconsider the decision-making process in 
light of external limitations dictated by criminal law. 

We have learned over the years that improving the human process of 
decision making requires criminal liability for negligence, and the same 
logic applies to al systems equipped with machine-learning functions. 
Although ai systems can be reprogrammed, their precious experience, 
gained through machine learning, would be lost. ar systems can learn the 
limits of their discretion on their own, but this is true for humans as well 
(and we must still impose criminal liability on humans for negligence 
offenses). 

In sum, if A1 entities are capable of fulfilling both the factual and mental 
element requirements of criminal liability for negligence offenses, and if 
the rationale for the imposition of criminal liability for these offenses is 
relevant to both humans and AI systems, there is no reason to avoid im- 
posing criminal liability in these cases. But this is not the only type of 
Al involvement in criminal liability for negligence offenses. 


3.3.2. h uman | iability: p erpetration-through-a nother 
and Semi- innocent a gents 
As described earlier, the most common way to deal with the instrumen- 
tal use of individuals in the commission of offenses is the general form 
of perpetration-through-another.*® To impose criminal liability for perpe- 
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tration of an offense through another, it is necessary to prove awareness 
of such instrumental use. Therefore, perpetration-through-another is ap- 
plicable only in the case of mens rea offenses. In most cases, the person 
being instrumentally used by the perpetrator is considered an innocent 
agent, and no criminal liability is imposed on him or her. The analysis 
of perpetration-through-another in the context of mens rea offenses has 
already been discussed. 

Nevertheless, a person instrumentally used can also be considered a 
semi-innocent agent who is criminally liable for negligence, although at 
the same time, the perpetrator is criminally liable for a mens rea offense. 
This is the case when negligence may be relevant for the perpetration- 
through-another, and that completes the discussion toward it. For exam- 
ple, a nurse in an operating room realizes that a person who had attacked 
her in the past is about to be operated on, and she decides that he deserves 
to die. She infects the surgical instruments with lethal bacteria, telling the 
surgeon that the instruments have been sterilized. 

A few hours after the surgery, the patient dies as a result of an infec- 
tion. Legal analysis of the case suggests that the nurse is the perpetrator- 
through-another of murder, having instrumentally used the surgeon to 
commit the murder. The surgeon’s criminal liability in this case depends 
on his mental state. If he is an innocent agent, he is exempt from criminal 
liability. But if he has the legal duty to make sure that the instruments have 
been sterilized, he is not an entirely innocent agent because he failed to 
fulfill his legal duties. 

At the same time, because he was not aware of the infected instru- 
ments, this is a case of negligence. When the agent is not aware of crucial 
elements of the offense, but a reasonable person in the same situation 
could and should have been aware, the agent is negligent and is called a 
semi-innocent agent.*® Thus, when a person instrumentally uses another 
person who is negligent regarding the commission of the offense, this is 
perpetration-through-another, but both persons are criminally liable: the 
perpetrator for a mens rea offense (for example, murder) and the other 
person for a negligence offense (for example, negligent homicide). Given 
that Ar systems are capable of forming negligence as a mental element, the 
question is whether they can function as semi-innocent agents. 

A case in which an ai system can be a semi-innocent agent is one in 
which the perpetrator (human, corporation, or Ar entity) instrumentally 
uses an AI system for the commission of an offense, and although it was 
used instrumentally, the al system was negligent with regard to commit- 
ting the offense. Only ar systems that are capable of fulfilling the mental 
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element requirement of negligence offenses can be considered as semi- 
innocent agents and can function as such. But not in every case in which 
the a1 system has the capability for negligence would it automatically func- 
tion as a semi-innocent agent. The capability for negligence is necessary 
for functioning as a semi-innocent agent, but it is not sufficient. 

The semi-innocent agent, whether human, corporation, or machine, 
must be examined ad hoc in any given case. Only if the agent was neg- 
ligent regarding the commission of the offense can it be considered a 
semi-innocent agent. Thus, if the instrumentally used a1 system did not 
consolidate awareness of the relevant factual data but had the capability 
to do so, and a reasonable person would have consolidated such aware- 
ness, the al system is considered a semi-innocent agent in the context of a 
perpetration-through-another (itself). 

The perpetrator’s criminal liability is not affected by the agent’s crim- 
inal liability, if any. The criminal liability of the perpetrator-through- 
another is for the mens rea offense, whether the instrumentally used 
AI system has no criminal liability (that is, is an innocent agent or lacks 
the relevant capabilities) or is criminally liable for negligence (that is, is a 
semi-innocent agent). Thus, using the legal construction of perpetration- 
through-another based on the instrumental use of an AI entity does not 
affect the status of the perpetrator as a mens rea offender, regardless of the 
AI system’s criminal liability. 

By the same token, the agent’s criminal liability in these cases is not 
directly affected by the perpetrator’s criminal liability. If the AT system 
was negligent (that is, it fulfilled both factual and mental element require- 
ments of the negligence offense), then criminal liability for negligence is 
imposed on it, and the system is classified as a semi-innocent agent in a 
perpetration-through-another. If the Ar system was not negligent because 
of its lack of capability or for any other reason, then no criminal liability 
is imposed on it, and the system is classified as an innocent agent in a 
perpetration-through-another. 

In other words, if the ar system is neither an innocent agent nor a 
semi-innocent agent, this means that it has fully met the requirements 
of the mens rea offenses. This is not the case for perpetration-through- 
another, but for principal perpetration on the part of the ar system. If the 
AI system is capable of being criminally liable for mens rea offenses as a 
sole offender, then there is nothing to prevent it from committing the of- 
fense jointly, with other entities, humans, corporations, or other Ar entities. 
Complicity by the AT systems requires at least mens rea, not negligence, 
because it requires awareness of the complicity and of the delinquent as- 
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sociation. This situation is not substantively different from meeting the 
requirements of any other mens rea offense, as discussed earlier.*” 


3.3.3. Joint h uman and ai entity | iability: 

probable c onsequence | iability 
Probable conseguence liability deals with the commission of an unplanned 
offense, different from or additional to the main, planned offense by more 
than one perpetrator. The guestion in these cases involves the criminal 
liability of the other parties to the unplanned offense committed by one 
of the parties. We have already discussed probable conseguence liability 
for an unplanned mens rea offense.” The relevant question here concerns 
probable consequence liability to an unplanned negligence offense com- 
mitted by an AI system. Are the programmers, users, and other related 
persons criminally liable for unplanned negligence offense committed by 
an AI system? 

This question winds up our two previous discussions regarding 
human criminal liability for negligence by ar entities in addition to or 
instead of the criminal liability of the Ar system for that offense.’ For ex- 
ample, a medical expert system is used to diagnose certain types of dis- 
eases by analyzing patient symptoms. The analysis is based on machine 
learning and inductive generalization of various cases. When the system 
fails to diagnose a case correctly, this results in the wrong treatment, lead- 
ing eventually to the patient’s death. Analysis of the system’s activity re- 
veals negligence by it, which meets both the factual and mental element 
requirements of negligent homicide. At this point a question arises about 
the programmer’s criminal liability for the offense. 

The programmer’s criminal liability has nothing to do with the deci- 
sion to use the al system, to follow its diagnosis, and so on, but involves 
the initial programming of the system. If the programmer had programmed 
the system to kill patients and had instrumentally used it for this purpose, 
this would have been perpetration-through-another of murder, but this is 
not the case here. In this case, the probable consequence liability may be 
relevant for the programmer’s criminal liability. 

The mental condition for the probable consequence liability requires 
the unplanned offense to be “probable” from the point of view of the party 
that did not actually commit it. It is necessary for that party to have been 
able to foresee and reasonably predict the commission of the offense. 
Some legal systems prefer to examine actual and subjective foreseeabil- 
ity (the party has actually and subjectively foreseen the occurrence of the 
unplanned offense), whereas others prefer to evaluate the ability to fore- 
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see through an objective standard of reasonability (the party has not actu- 
ally foreseen the occurrence of the unplanned offense, but any reasonable 
person in this situation could have). 

Actual foreseeability parallels subjective mens rea, whereas objec- 
tive foreseeability parallels objective negligence. Conseguently, in legal 
systems that reguire objective foreseeability, the programmer should be 
at least negligent for the commission of the negligence offense by the 
Al system for the imposition of criminal liability for that offense. But in 
legal systems that require subjective foreseeability, the programmer must 
be at least aware of the possibility of the commission of the negligence of- 
fense by the a1 system for imposition of criminal liability for that offense. 

However, if neither subjective nor objective foreseeability of the com- 
mission of the offense apply to the programmer, the probable consequence 
liability is irrelevant. In this case, no criminal liability can be imposed on 
the programmer, and the criminal liability of the AT system for the negli- 
gence offense does not affect the programmer’s liability. 


Closing The opening exaMple: negligent Killin g robot 


We opened this chapter with an example describing an expert system that 
killed through negligence. We can now return to this case and analyze it 
based on insights achieved in this chapter. There are four main charac- 
ters whose criminal liability can be considered for this homicide: the a1 
system, the programmer, the hospital, and the user of the Ar system. To 
examine the criminal liability of the al system, we must examine first its 
technological capabilities. If the a1 system has the technological capabili- 
ties to meet the requirements of negligence offenses, it forms a machina 
sapiens criminalis in this context. 

If these capabilities are present, the AI system’s criminal liability 
should be examined as if it were human. But if the ai system does not 
have the technological capabilities to meet the requirements of negligence 
offenses, it cannot be considered a machina sapiens criminalis and is not 
criminally liable for the negligent homicide. In the opening example, the 
medical expert system met the factual element requirement for negligent 
homicide because its diagnosis and recommendations set off a chain of 
events that ended with the baby’s death.*° 

To meet the mental element requirement, it is necessary to prove neg- 
ligence on the part of the system. To do so, it is required to show that 
the system has the capability to consolidate awareness, that it did not in 
practice form awareness of the relevant factors involved in the mistaken 
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diagnosis, and that a reasonable person could have formed such aware- 
ness. The first condition can be proven based on the characteristics of the 
system and according to the definition of awareness in criminal law (per- 
ception by the senses of factual data and its understanding), as discussed 
earlier.” If the system has such capability, and if that capability has been 
activated, then the system formed awareness. 

If the system had been aware of the bacterial infection and nevertheless 
produced an inappropriate diagnosis and recommendations, this would 
not have been negligence but mens rea, and the homicide offense would 
have been manslaughter or murder. Otherwise, the system meets the 
second condition for negligence. For the ar system to be criminally liable 
for negligent homicide, its mental state must be compared with that of a 
reasonable person. The subjective incarnation of the reasonable person in 
this case is a reasonable medical expert system. The question is whether 
the system took into consideration all the relevant factors required for 
proper diagnosis, and whether they were appropriately weighed, given the 
relevant circumstances of the case. 

The diagnosis produced by the system did not necessarily have to be 
correct for the system to be exempt from liability for negligence. Not every 
incorrect diagnosis is considered negligent. At issue here is the system’s 
judgment. If it is proven that the system’s judgment was correct (the cal- 
culation took all factors into consideration and weighed them properly), 
then the system acted reasonably. Although the diagnosis did not cor- 
rectly identify the cause of the symptoms, this is still not in the realm of 
negligence if the system’s judgment was reasonable. Not every mistake is 
considered negligent. In this case, the court would have to examine cir- 
cumstances carefully, with or without the help of medical and software 
experts. 

If the court reaches the conclusion that a reasonable person would 
have been aware of the possibility that the symptoms were caused by a 
bacterial infection, or would have considered the probability to be low that 
influenza had caused the symptoms, then the system is considered negli- 
gent. Otherwise, the system is exempt from criminal liability for the death 
of the baby. Three other actors may be criminally liable in this case, in 
addition to (but not instead of) the ar system. The criminal liability of the 
AI system may affect that the liability of these actors, but not necessarily. 

The programmer’s criminal liability is determined by his role in the oc- 
currence of the death. If the programmer instrumentally used the al system 
to kill the baby by designing it in such a way as to produce erroneous 
medical diagnoses, the programmer is a perpetrator-through-another of 
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the homicide. This would also be the case when the A1 system had no ad- 
vanced al capabilities to meet the requirements of the offense, so that the 
programmer and the other related parties, but not the AI system, are the 
only ones criminally liable for the homicide. Their exact criminal liabil- 
ity for the homicide, whether murder or manslaughter, is determined by 
whether they meet the mental element of these offenses. 

If the programmer has foreseen the consequence (the patient’s death), 
or if any reasonable programmer could have foreseen it, and if the a1 system 
is criminally liable for the homicide, the programmer may be criminally 
liable for negligent homicide through probable consequence liability, if the 
offense was unplanned. Generally, if the programmer did not intention- 
ally design the system to commit any offense, and an unplanned homi- 
cide (murder, manslaughter, or negligent homicide) occurred, the system’s 
criminal liability is examined based on its capabilities and on meeting the 
requirements of the relevant homicide, and the programmer’s criminal li- 
ability is examined based on the standards of negligence. Here, program- 
mer refers not only to the person or persons who actually programmed the 
system, but to all related persons, including the company that produced 
the system. 

Two other actors who may be criminally liable are the users of the 
system (the medical staff) and the hospital. The hospital may be criminally 
liable for making the decision to use an A1 system in real-life medical cases. 
To determine the liability of the hospital, it is necessary to examine the 
factual data about the system that was available and known to the hospital, 
including the probability that the system would make mistakes. If the deci- 
sion was reasonable, the hospital is not negligent. Similarly, the medical 
staff must be examined to determine its criminal liability for homicide. 
The factual data about the system that was available and known to the staff 
is extremely important in evaluating the decision of the medical staff to 
follow the system’s diagnosis and recommendations. 

If the ar system was correct in the previous 1,000 cases, then the medi- 
cal staff’s decision to follow its recommendations may be reasonable. But 
it is up to the court to determine the sphere of reasonability in each case. 
Perhaps in emergency rooms a 10 percent rate of mistakes is reasonable, 
whereas in cosmetic surgery such a rate is far beyond reasonability. Thus, 
criminal liability of ar systems for negligence offenses is possible, and it 
does not necessarily reduce the criminal liability of other relevant entities. 


ai criminal lia Bili Ty for 
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The operations carried out by drones are based on artificial intelligence 
technology. A1 drones operate not only on the ground (unmanned vehi- 
cles), but also in the air and underwater.’ For example, in 2012 the US 
Navy examined an operative drone that was slated to land on an aircraft 
carrier, relying on pinpoint cpr coordinates and advanced avionics. Com- 
puters on the carrier digitally transmit the relevant data (speed, cross- 
winds, and so on) to the drone, as it approaches for the landing. The drone 
not only knows how to land by itself, but it also knows the weapons it is 
carrying; if, when, and where it needs to refuel from an aerial tanker; and 
whether there is a nearby threat. It does its own calculations and decides 
what to do next.? 

The same technology is implemented in other devices as well, with 
the possible result that these devices can commit traffic offenses, whether 
in the air or on the ground. For example, an unmanned vehicle is used for 
ground transportation on roads travelled by other, human-driven vehicles. 
When the unmanned vehicle arrives at an intersection, it analyzes the 
traffic situation and decides to proceed. The vehicle has a green light, and 
the lights in all other directions are red, but an ambulance arrives at high 
speed and crosses the intersection. 

The vehicle hits the ambulance, causing severe injury to the passen- 
gers, and the patient in the ambulance dies as a result. Analysis of the pa- 
tient’s medical situation shows that the patient would have survived had 
the ambulance not been hit by the unmanned vehicle, so the collision is 
the direct factual cause of the patient’s death. The legal question is, who is 
to be held criminally liable for the patient’s death? 

Strict liability offenses are in common use by most countries in a con- 
text of public order and welfare (for example, traffic offenses, labor safety 
offenses, or environmental offenses). For imposition of criminal liability 
in these offenses, both the factual and mental element requirements must 
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be met by the apparent offender. Conseguently, the guestion regarding 
the criminal liability of AI entities for strict liability offenses is whether 
they are capable of meeting these requirements accumulatively. Moreover, 
even if these entities have such capability, this does not serve to exempt 
humans from criminal liability if they were involved in the commission 
of the offense. 

To explore these issues, we examine the capability of ai entities to 
meet both the factual and mental element requirements of strict liability 
offenses, and the criminal liability of human offenders for strict liability 
offenses committed by ai entities. 


4.1. The facTual eleMenT r eQuire MenT 


The factual element requirement structure (actus reus) is identical for 
all types of offenses: intentional, negligence, and strict liability, as noted 
earlier." This structure contains one mandatory component (conduct) for 
all offenses, and two optional components (circumstances and results) 
for some offenses. The capability of machines to meet the factual element 
requirement, considering this structure, has already been discussed, 
and the conclusions of that discussion hold true for negligence offenses 
as well. 


4.2. The Mental eleMenT r eQuire MenT 


Imposition of criminal liability for strict liability offenses requires meet- 
ing both factual and mental elements requirements. The mental element 
requirement of strict liability offenses is strict liability or presumed neg- 
ligence. If AT technology is capable of meeting the strict liability require- 
ment, it is possible and feasible to impose criminal liability on it for strict 
liability offenses. We begin by exploring the structure of strict liability 
requirements, and then proceed to examine the way in which ar technol- 
ogy meets the requirement. 


4.2.1. Structure of Strict | iability r equirement 
Strict liability evolved from absolute liability and was accepted in criminal 
law as a form of mental element requirement. Beginning in the eighteenth 
century, English common law determined that some offenses require nei- 
ther mens rea nor negligence. These offenses, called public welfare of- 
fenses, were inspired by tort law, which accepted absolute liability as legit- 
imate.” Consequently, these became criminal offenses of absolute liability, 
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and imposition of criminal liability for them required proof of the factual 
element alone. 

Absolute liability offenses were considered exceptional because they 
require no mental element. In some cases Parliament intervened and re- 
quired a mental element,’ and in other cases mental element requirements 
were added by court rulings. By the mid-nineteenth century, English 
courts began to consider efficiency criteria as part of criminal law in vari- 
ous contexts, which gave rise to convictions on the basis of public incon- 
venience. Offenders were indicted and convicted despite the fact that no 
mental element was proven because of the public inconvenience caused 
by the commission of the offense.’ 

These convictions created, in practice, an upper threshold for neg- 
ligence, a type of augmented negligence, which required individuals to 
follow strict guidelines and make sure they committed no offenses. This 
standard of behavior is higher than the one imposed by negligence, which 
requires only reasonable conduct. Strict liability offenses require more 
than reasonability, making sure that no offense is committed at all. These 
offenses show a clear preference of the public welfare over the strict jus- 
tice for the potential offender. Because these offenses were not considered 
severe, they were expanded “for the good of all.” 

This development was considered necessary because of the legal and 
social changes at the time of the first industrial revolution. For example, 
the increasing number of workers in the cities caused employers to de- 
grade the workers’ social conditions. Parliament intervened through social 
welfare legislation, and the efficient enforcement of this legislation was 
through absolute liability offenses." It was immaterial whether the em- 
ployer knew what the proper social conditions for workers were; employ- 
ers had to make sure that no violation of these conditions occurred.’ In the 
twentieth century, this type of criminal liability spread to other spheres of 
law as well, including traffic law. 

American criminal law accepted absolute liability as a basis for crimi- 
nal liability in the mid-nineteenth century,” ignoring previous rulings that 
rejected it.” Acceptance was restricted to petty offenses, and violations 
were punished by fines, and not very severe ones at that. Similar accep- 
tance occurred at about the same time in the European-Continental legal 
systems, so that absolute liability in criminal law became a global phe- 
nomenon. In the meantime, the fault element in criminal law became 
much more important because of internal developments in criminal law, 
and mens rea became the major and dominant requirement for mental ele- 
ment in criminal law. 
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Therefore, criminal law was reguired to make changes in absolute li- 
ability in order to meet the modern understanding of fault, which triggered 
the move from absolute to strict liability. At the heart of the change was 
the move from absolute legal presumption (praesumptio juris et de jure) to 
relative legal presumption (praesumptio juris tantum), so that the offender 
had the opportunity to refute the criminal liability. The presumption was 
presumption of negligence, either refutable or not.” The move from abso- 
lute liability to strict liability eased the acceptance of the presumed negli- 
gence as yet another, third form of mental element in criminal law. 

Since the broad acceptance of strict liability worldwide, legal systems 
have justified it both from the perspective of fault in criminal law*® and 
constitutionally. The European Court of Human Rights justified using strict 
liability in criminal law in 1998,” and thus strict liability was considered 
not to be contradicting the presumption of innocence, protected by the 
1950 European Human Rights Covenant.” This ruling has been embraced 
in Europe and in Britain. The Supreme Court of the United States ruled 
consistently that strict liability does not contradict the US Constitution.” 
So did the supreme courts in various states.” But in the United States, the 
courts recommended to restrict the use of these offenses to the minimum 
necessary, and to prefer using mens rea or negligence offenses. 

The strict liability construct in criminal law focuses on the relative 
negligence presumption and the ways to refute it. According to this pre- 
sumption, if all components of the offense’s factual element requirement 
are proven, it is presumed that the offender was at least negligent. Conse- 
quently, for the imposition of criminal liability in strict liability offenses, 
the prosecution does not have to prove the mental state of the defendant, 
only the fulfillment of the factual element requirements. The mental state 
of the offender is derived from the conduct. Up to this point, strict liability 
is similar to absolute liability. 

But contrary to absolute liability, strict liability can be refuted by the 
defendant because it is based on a relative legal presumption. To refute 
strict liability, the offender must accumulatively meet two conditions: 
(1) no mens rea or negligence existed on the part of the offender, and 
(2) all reasonable measures to prevent the offense were taken. 

The first condition refers to the mental state of the offender. According 
to the presumption, the commission of the factual element presumes that 
the offender was at least negligent. This means that the offender’s mental 
state was one of negligence or of mens rea. The offender must therefore 
refute the conclusion of the presumption, and prove it incorrect in his 
case. To do so, the offender must prove that he was not aware to the rele- 
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vant facts, and that no other reasonable person could have been aware of 
them under the same circumstances. This proof resembles refuting mens 
rea in mens rea offenses and negligence in negligence offenses. 

However, strict liability offenses are not mens rea or negligence of- 
fenses, because refuting mens rea and negligence is not sufficient to pre- 
vent imposition of criminal liability. The social and behavioral purpose 
of these offenses is to ensure that individuals conduct themselves strictly 
according to rules, and commit no offenses. That should be proven as well. 
The offender must therefore prove that he or she took all reasonable mea- 
sures to prevent the offense.”* Note the difference between strict liability 
and negligence: to refute negligence, it is sufficient to prove that the of- 
fender acted reasonably, but to refute strict liability, it is necessary to prove 
that all reasonable measure were taken.” 

To refute the negligence presumption of strict liability, the defendant 
must positively meet both conditions by a preponderance of the evidence, 
as in civil law cases. The defendant is not required to prove that these 
conditions have been met beyond any reasonable doubt, but in general, it 
is not sufficient to merely raise a reasonable doubt. The burden of proof 
is higher than the general burden of the defendant. The possibility for the 
offender to refute the presumption becomes part of the strict liability re- 
quirement because it relates to the offender’s mental state. The structure 
of strict liability is described schematically in table 4.1. 

The modern structure of strict liability is consistent with the concept 
of criminal law as a matrix of minimal requirements, described earlier,”® 
and it has both internal and external aspects. Internally, strict liability is 
the minimal mental element requirement for each of the factual element 
components. Consequently, if strict liability is proven in relation to cir- 
cumstances and results but negligence is proven in relation to conduct, 
that satisfies the requirement of strict liability. This means that for each 
of the factual element components, at least strict liability is required, but 
not exclusively strict liability. Externally, the mental element require- 
ment of strict liability offenses is satisfied by at least strict liability, but 
not exclusively. This means that criminal liability for strict liability of- 
fenses may be imposed by proving mens rea and negligence as well as 
strict liability. 

Because strict liability is still considered an exception to the general 
requirement of mens rea, it is an adequate mental element for relatively 
light offenses. In some legal systems, strict liability has been restricted ex 
ante or ex post to light offenses.” This general structure of strict liability 
is a template that contains terms derived from the mental terminology. To 
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Table 4.1 - General Structure of Strict Liability 


fac TUal elemen T componen T STric T lia Bili Ty componen T 


Conduct The conduct has been committed, 
and the offender fails to prove that 
neither mens rea nor negligence 
requirements for the conduct have 
been met, and that all reasonable 
measures to prevent the offense were 
taken. 


Circumstances The circumstances are present, 

and the offender fails to prove that 
neither mens rea nor negligence 
requirements for the conduct have 
been met, and that all reasonable 
measures to prevent the offense were 
taken. 


Results The results have occurred, and the 
offender fails to prove that neither 
mens rea nor negligence require- 
ments for the conduct have been 
met, and that all reasonable mea- 
sures to prevent the offense were 
taken. 


explore whether Ar entities are capable of meeting the strict liability re- 
quirement of given offenses, we must examine these terms. 


4.2.2. f ulfillment of Strict | iability: 
making the f actual Become mental 
To prove that the defendant met the requirement for strict liability, the 
prosecution may choose to prove only the factual element of an offense. 
According to the negligence presumption, the presence of the factual ele- 
ment indicates that the offender has been at least negligent. But as already 
noted,” the possibility of refuting that presumption by the defendant is 
an integral part of the substance and structure of strict liability. There- 
fore, strict liability may apply only to offenders who possess the mental 
capability of refuting the presumption, which has to do with the mental 
element. 
The mental capability of refuting the presumption does not necessarily 
mean that the offender has proof of his or her innocence, or has convinc- 
ing arguments to that effect, but only that the inner capabilities required 
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to refute the negligence presumption are present. Refuting the presump- 
tion reguires proof that the offender (1) was neither aware nor negligent 
regarding the factual element components, and (2) has taken all reasonable 
measures to prevent the commission of the offense. The reguired capabili- 
ties, therefore, are those needed to consolidate awareness and negligence, 
and those needed to act reasonably. 

The guestion is whether Ar systems possess such capabilities, and 
therefore whether criminal liability for strict liability offenses may be 
imposed on them. We examine these capabilities one by one. First is the 
capability to consolidate awareness and negligence. We have examined 
this capability of ar systems already in relation to mens rea and negli- 
gence.” al systems that possess the relevant features discussed earlier do 
indeed have this capability. Consequently, all ar systems that can be in- 
dicted for mens rea and negligence offenses can also be indicted for strict 
liability offenses. 

This makes sense. Analogously, if mens rea and negligence require 
higher mental capabilities than does strict liability, then the lower capa- 
bility required for strict liability would be much easier to achieve than the 
higher capabilities needed for mens rea and negligence. Therefore, the 
Al offender is required to possess the same features, regardless of the 
type of offense. Because negligence requires the capability to consolidate 
awareness,”° and because strict liability requires the capabilities to consol- 
idate awareness and negligence, it is clear that all a1 offenders are required 
to possess the capability of consolidating awareness. 

So in this context, an indictable offender is one who has the capability 
of consolidating awareness, regardless of whether this capability has been 
realized and utilized, and regardless of the type of offense. An AI system 
that is indictable for strict liability offenses must have the mental features 
and capabilities needed to be indicted for mens rea or negligence offenses. 
This fact reveals the character of the Ar offender, with respect to inner ca- 
pabilities, as more or less uniform. Thus, the minimal inner features and 
capabilities of machina sapiens criminalis are uniform and can be defined 
accordingly. 

The second required capability is reasonability. Refuting the negli- 
gence presumption requires proof of having taken all reasonable measures, 
as already noted, and the capability of acting reasonably. Without such ca- 
pability, no A1 system can take all reasonable measures. The capability for 
reasonability is required in negligence as well. This is the same capability 
as the one required for strict liability, and we have discussed it already in 
the context of negligence.*! The fact that the same capability is required 
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strengthens the argument just made that all inner capabilities of machina 
sapiens criminalis are uniform, regardless of the type of offense. 

Although the capability is the same in both negligence and strict li- 
ability offenses, it operates differently in the cases of negligence and strict 
liability. For negligence offenses, it is required that the a1 system identify 
the reasonable options and make a choice between them. The practical re- 
quirement is that its final choice be a reasonable option. For strict liability 
offenses, it is also required that the ar system identify the reasonable op- 
tions, but in addition, it must choose to exercise all of them. The choice 
here is between reasonable and unreasonable, whereas in the case of neg- 
ligence, the choice is among the reasonable options. 

It is much easier for an al system, under these conditions, to act reason- 
ably in a strict liability context than in the context of negligence because 
in strict liability situations, fewer choices are required. Accordingly, in 
relation to the criminal liability of an a1 system in strict liability offenses, 
the court must answer the following three questions: 


1. Was the factual element of the offense fulfilled by the ai system? 

2. Does the a1 system have the general capability of consolidating 
awareness or negligence? 

3. Does the al system have the general capability of reasonability? 


If the answer to all three questions is affirmative, and this is proven beyond 
any reasonable doubt, then the a1 system has fulfilled the requirements of 
the strict liability offense, and the A1 system is presumed to be at least 
negligent. 

At this point, the defense has the opportunity to refute the negligence 
presumption through positive evidence. After the evidence is presented, 
the court must decide in two questions: 


1. Has the Ar system indeed formed mens rea or negligence re- 
garding the factual element components of the strict liability 
offense? 

2. Has the ai system not taken al] reasonable measures to prevent 
the actual commission of the offense? 


If the answer to even one of these two questions is affirmative, the neg- 
ligence presumption has not been refuted, and criminal liability for the 
strict liability offense is imposed. Only if the answers to both questions are 
negative is the negligence presumption refuted, and no criminal liability 
for the strict liability offense can be imposed on the a1 system. In general, 
AI systems that are capable of forming awareness and negligence, as dis- 
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cussed earlier,” have neither technological nor legal limitations in forming 
the inner requirements for strict liability offenses because strict liability 
requires a lower level of mental element than do mens rea or negligence. 
Given that strict liability is relevant to ar technology and that it is possible 
to prove it in court, then who is to be criminally liable for the commission 
of this type of offenses? 


4.3. Cri Minally liable entities for ai sTricT liabili Ty offenses 


In general, imposition of criminal liability for strict liability offenses re- 
quires meeting both factual and mental elements of these offenses. Humans 
are involved in the creation, design, programming, and operation of 
AT technology and entities. Consequently, when the factual and mental ele- 
ments of the offense are met by the a1 entity, the question is, who is to be 
criminally liable for the offenses committed? The possible answers are the 
Al entity, the humans, or both. We will now discuss these answers. 


4.3.1.ai entity | iability 

In strict liability offenses, such as mens rea and negligence, when the of- 
fender fulfills both the factual and mental element requirements of the 
offense, criminal liability is imposed. As in the case of mens rea and neg- 
ligence offenses, the court is not supposed to check whether the offender 
had been “evil” or “immoral,” as already discussed.** This is true for all 
types of offenders, including humans, corporations, and a1 entities. There- 
fore, the justifications used to impose criminal liability on AT entities in 
mens rea and negligence offenses are relevant for strict liability as well. 
As long as these requirements are narrowly met, criminal liability can 
be imposed. 

But strict liability and negligence offenses are different from mens rea 
offenses in their social purpose, as we have discussed.** The relevant ques- 
tion is whether their different social purpose applies not only to humans 
and corporations, but to AI entities as well. From their inception, strict 
liability offenses were not designed to deal with evil people, but with 
individuals who did not make all possible efforts to prevent the commis- 
sion of an offense. Therefore, the debate about evil in criminal law is not 
relevant to strict liability offenses in the same way that it may be relevant 
for mens rea offenses. 

In this context, criminal law function is to educate and ensure that no 
offense is committed, and strict liability offenses serve to draw the bound- 
aries of individual discretion. Any person can use wrong judgment and 
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make some, but not all possible efforts to prevent the commission of an 
offense. Most of the time, this does not contradict the norms of criminal 
law. For example, we may take the risk of investing our money in doubtful 
stocks, and we may not take all possible precautions to prevent damage to 
our investments, but that does not breach any criminal law. 

In some cases, however, wrong judgment contradicts a norm of crimi- 
nal law that was designed to educate individuals to make sure that no 
offense is committed. As long as our wrong judgment and lack of effort 
to prevent offenses do not contradict the criminal law, society expects us 
to learn our lesson on our own. Next time we invest our money, we will 
examine the details of our investment more carefully. This is how we gain 
life experience. In this case, society takes a risk that we will not learn our 
lesson, but still does not intervene through criminal law. 

But when criminal offenses are committed, society does not assume 
the risk of allowing individuals to learn their lessons on their own. In 
these cases the social harm is too grave to allow such risk, and society 
intervenes using both strict liability and negligence offenses. In the case 
of strict liability offenses, the purpose is to educate individuals to make 
every possible effort to prevent the occurrence of the offense. For example, 
drivers are expected (and educated) to drive carefully to prevent any pos- 
sible moving violation. 

The objective of the offense is to make sure that individuals learn how 
to behave extra carefully. Prospectively, it is assumed that after some- 
one has been convicted of a strict liability offense, the probability for re- 
commission of the offense is much lower. For example, society educates 
its drivers to drive extra carefully, and its employers to adhere to state reg- 
ulations regarding the payment of wages, and so on. Human and corporate 
offenders are expected to learn how to behave carefully by means of the 
criminal law. Is this method of education relevant for a1 entities as well? 

The answer is affirmative. For the educational purpose of strict liabil- 
ity, there is not much use in imposing criminal liability unless the offender 
has the ability to learn and change his or her behavior accordingly. If we 
want to make offenders learn to behave very carefully, we must assume 
that they are capable of learning and implementing that knowledge. If such 
capabilities are present and exercised, it is necessary to impose criminal 
liability for strict liability offenses. But if no such capabilities are exer- 
cised, then imposing criminal liability is entirely unnecessary because no 
prospective value is expected, and the result of using or not using criminal 
liability for strict liability offenses would be the same. 

For al systems that are equipped with the relevant capabilities for ma- 
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chine learning, criminal liability for strict liability offenses is necessary 
if these capabilities are to be applied in the relevant situations in which 
there is an obligation to behave extra carefully. Exactly as for humans, 
strict liability offenses may trace the boundaries of discretion for AT sys- 
tems. Humans, corporations, and AI systems alike are supposed to learn 
from their experience and improve their decisions prospectively, includ- 
ing the standards of carefulness. 

When the absence of carefulness is part of criminal law, the law inter- 
venes to shape the judgment exercised with respect to careful behavior. 
For the A1 system, the criminal liability for strict liability offenses is an 
opportunity to reconsider the decision-making process in light of external 
limitations dictated by criminal law, which require extra-careful conduct 
and decision making. We have learned over many generations that the 
human decision-making process requires criminal liability for strict liabil- 
ity in order to be improved, and the same logic applies to al systems using 
machine learning methods. 

Naturally, ar systems can simply be reprogrammed each time, but this 
is different from the precious experience gained through machine learn- 
ing. Al systems may be capable of learning their boundaries and correcting 
their judgment on their own, but the same is true for humans, and we still 
impose criminal liability on humans for strict liability offenses. Conse- 
quently, if ar entities have the required capabilities for meeting both fac- 
tual and mental elements of criminal liability for strict liability offenses, 
and if the rationale for the imposition of criminal liability for these of- 
fenses is relevant to both humans and A1 systems, then there is no reason 
to avoid imposing criminal liability in all these cases. But this is not the 
only type of ar involvement in criminal liability for strict liability offenses. 


4.3.2. h uman | iability: p erpetration-through-a nother 

As described earlier, the most common way of handling the instrumen- 
tal use of individuals for the commission of offenses is the general form 
of perpetration-through-another. To impose criminal liability for the per- 
petration of an offense-through-another, it is necessary to prove aware- 
ness of the instrumental use. Therefore, perpetration-through-another is 
applicable only in mens rea offenses. In most cases, the person being in- 
strumentally used by the perpetrator is considered an innocent agent, and 
no criminal liability is imposed on him or her. We have already analyzed 
perpetration-through-another in a context of mens rea offenses. 

The person who is instrumentally used, however, may also be con- 
sidered a semi-innocent agent who is criminally liable for negligence, al- 
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Table 4.2 - The Legal State Reflected in the Criminal Mental 
State of the Other Person in Perpetration-through-Another 


criminal men Tal STaTe 


of The oTher per Son legal STaTe of The oTher per Son 


mens rea accomplice 
negligence semi-innocent agent 
strict liability innocent agent 
none innocent agent 


though the perpetrator is at the same time criminally liable for a mens rea 
offense. Negligence is the lowest level of mental element required for the 
person instrumentally used to be considered a semi-innocent agent. In this 
context, strict liability is too low a level of mental element for consider- 
ation as a semi-innocent agent. If the person who is instrumentally used by 
another is in a mental state of strict liability, that person is to be considered 
an innocent agent, as if lacking any criminal mental state at all. 

Thus, in perpetration of an offense-through-another, the other (instru- 
mentally used) person may be in four possible mental states that reflect 
matching legal consequences, as described schematically in table 4.2. 

When the other person is aware of the delinquent enterprise and still 
continues to participate although he is under no pressure, he becomes an 
accomplice to the commission of the offense. Negligence reduces the per- 
son’s legal state to that of a semi-innocent agent.“ But whether the mental 
state of this person is that of strict liability or an absence of a criminal 
mental state, he must be considered an innocent agent, so that no criminal 
liability is imposed on him, and the full criminal liability for the rele- 
vant offense is imposed on the perpetrator who instrumentally used that 
person. This is true for both humans and AT entities. 

For this legal construct of perpetration-through-another, it is immate- 
rial whether the al system was instrumentally used, and whether it used 
its strong al capabilities to form a strict liability mental state. Thus, the 
instrumental use of a weak ar system, of a strong al system that formed 
strict liability, or of a screwdriver has the same legal consequences. The 
entity making the instrumental use (human, corporation, or AI system) is 
criminally liable for the commission of the offense in full, and the instru- 
mentally used entity (human, corporation, or AT system) is considered an 
innocent agent. 

For example, the human user of an unmanned vehicle based on an ad- 
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vanced al system instrumentally uses the A1 system to cross an intersection 
where the traffic light is red. The system has the capability to be aware of 
the traffic light’s color, but in practice it is not aware of the red light. The 
system does not take all reasonable measures to examine the data, such as 
analyzing the color of the traffic light facing in its direction. Crossing an 
intersection on a red light is a strict liability offense. In this instance, the 
human is criminally liable for that offense as perpetrator-through-another. 
Because the ar system was instrumentally used by the perpetrator, it is 
considered an innocent agent, and therefore criminal liability is not im- 
posed on it. 


4.3.3. Joint human and ai entity | iability: 
probable c onsequence | iability 

Probable consequence liability addresses the commission of unplanned 
offenses (different from a planned offense or additional to it). The ques- 
tion in these cases is about the criminal liability of the other parties to 
the unplanned offense committed by one party. Previously, we discussed 
probable consequence liability for unplanned mens rea and negligence of- 
fenses.*” The relevant question here is regarding the probable consequence 
liability for unplanned strict liability offenses committed by an AT system. 
Are the programmers, users, and other related persons criminally liable for 
unplanned strict liability offense committed by an AT system? 

For example, two people are committing a bank robbery. To escape 
from the arena, they use an unmanned vehicle based on an advanced 
Al system. During their escape, the vehicle exceeds the legal speed limit, 
committing a strict liability offense. Analyzing the records of the vehicle 
shows that it meets the strict liability requirements of the offense. The 
bank robbers did not program the vehicle to commit the offense, and did 
not order it to do so. The question concerns the robbers’ criminal liabil- 
ity for the strict liability traffic offense committed by the vehicle, in ad- 
dition to their criminal liability for robbery. If the offenders had ordered 
the vehicle to drive at that speed and had instrumentally used it for that 
purpose, this would have been perpetration-through-another, but that is 
not the case here. 

Probable consequence liability may be relevant here for the criminal 
liability of the bank robbers. The mental condition for probable conse- 
quence liability requires the unplanned offense to be “probable” from the 
point of view of the party that did not commit it. In other words, it is nec- 
essary for that party to have been able to foresee and reasonably predict the 
commission of the offense. Some legal systems prefer to examine actual 
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and subjective foreseeability (the party has actually and subjectively fore- 
seen the occurrence of the unplanned offense), whereas others prefer to 
evaluate the ability to foresee based on an objective standard of reasonabil- 
ity (the party has not actually foreseen the occurrence of the unplanned 
offense, but any reasonable person under the same condition would have). 

Actual foreseeability parallels subjective mens rea, whereas objective 
foreseeability parallels objective negligence. A lower level of foreseeabil- 
ity is not adeguate for probable conseguence liability. Thus, the guestion 
is about the level of foreseeability reguired of the robbers. If they actually 
foresaw the commission of that offense by the vehicle, then criminal li- 
ability for the offense is imposed on them in addition to criminal liability 
for robbery. If the robbers formed objective foreseeability, then criminal 
liability for the additional offense is imposed only in legal systems in 
which probable conseguence liability can be satisfied through objective 
foreseeability. 

But if the mental state of the bank robbers in relation to the additional 
offense is strict liability, this is not adeguate for the imposition of criminal 
liability through probable conseguence liability. Although the additional 
offense reguires strict liability for imposition of criminal liability, this is 
true only for the perpetrator of that offense, and not for imposing criminal 
liability through probable consequence liability. Thus, if neither subjective 
nor objective foreseeability regarding the commission of the offense can be 
attributed to the robbers, then probable consequence liability is irrelevant. 
In this type of cases, no criminal liability is imposed on the offenders, and 
the criminal liability of the at system for the strict liability offense does 
not affect the offenders’ liability. 


Closing The opening exaMple: sTric T liable Killin g r oboT 


We opened this chapter with an example of a killing robot, an unmanned 
vehicle based on an al system, subject to strict liability. Returning to the 
example, we can now analyze it using the insights gained from this chap- 
ter. The criminal liability of three main characters must be considered for 
the homicide in this example: the ai system, the manufacturer (including 
the programmers), and the user. To examine the criminal liability of the 
Al system, first we must investigate its technological capabilities. If the 
AI system has the technological capabilities needed to meet the require- 
ments of strict liability offenses, it forms a machina sapiens criminalis in 
this context. 

If such capabilities are present, the criminal liability of the a1 system 


When r obo Ts Kill - 118 


should be examined as if it were human. But if the ar system lacks the 
technological capabilities needed to meet the requirements of strict liabil- 
ity offenses, it does not form a machina sapiens criminalis in this context 
and has no criminal liability for the offense. In the opening example, the 
AI system met the factual element requirement of homicide, because its 
conduct caused a chain of events that ended in the patient’s death.% 

To meet the mental element requirement, we must prove strict liabil- 
ity, which includes proving that the AT system is capable of consolidat- 
ing awareness or negligence, and is capable of reasonability. To refute the 
negligence presumption, it is necessary to prove that the al system has not 
actually formed awareness of relevant facts or negligence toward them, 
and has taken all reasonable measures to prevent the commission of the 
offense. The first condition may be proven based on the characteristics of 
the system, using the definitions of awareness and negligence in criminal 
law discussed earlier.” If the system possesses such capability, and if such 
capability has been activated, the system formed awareness or negligence. 

The capability for reasonability is not different from the required par- 
allel capability for negligence, previously discussed.*° The system is not 
required to act reasonably, only to possess the capability of making rea- 
sonable decisions. Strong AI systems, which can calculate the probabili- 
ties of the occurrence of events based on current factual data (such as 
weather forecast computers*') and choose options that meet certain crite- 
ria, are supposed to have this type of capability. If no additional evidence 
is brought before the court, this is sufficient to activate the negligence pre- 
sumption and to impose criminal liability for the strict liability offense. 

Refuting the negligence presumption begins with positive proof of not 
having formed awareness or negligence. This may be proven based on the 
records of the ai system. If proven, it is necessary to prove having taken all 
reasonable measures for the prevention of the offense. Because the capabil- 
ity for reasonability has already been proven, the conduct of the system is 
examined accordingly. The reasonable options calculated by the system 
and the actions it took based on these calculations must be examined to 
determine whether they include all the reasonable options in this case. 
The examination is carried out the same way as in the case of negligence, 
at times using expert witnesses. If it is proven that no actual awareness or 
negligence were formed regarding the factual data, and that all reasonable 
measures have been taken by the system, the negligence presumption is 
refuted. If not, criminal liability for the strict liability offense is imposed. 

The criminal liability of the manufacturer, the programmer, and the 
user is determined based on their role in causing the death. If they instru- 
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mentally used the ai system to kill the patient in the ambulance by design- 
ing the system in such a way that it would collide with the ambulance 
because it misunderstands the factual situation, they are perpetrators- 
through-another of homicide. This is also the case if the AT system has 
no advanced ai capabilities to meet the requirements of the offense. In 
this case, they and the related parties, but not the ar system, are the only 
ones criminally liable for the homicide. Their exact criminal liability for 
the homicide, whether as murder or manslaughter, is determined by their 
meeting the mental element requirements of these offenses. 

If the manufacturer, the programmer, and the user foresaw the conse- 
quences (the patient’s death) or any reasonable person in their condition 
could have foreseen these consequences, and if the a1 system is criminally 
liable for the homicide, they may be criminally liable for the homicide 
through probable consequence liability, as long as the offense was un- 
planned. Generally, if they did not design the system to commit any of- 
fense, but the unplanned homicide occurred, then the criminal liability of 
the system is examined based on its capabilities and on meeting the rel- 
evant homicide requirements, and the criminal liability of the human par- 
ties is examined based on the standards of probable consequence liability. 

Therefore, al systems can be criminally liable for strict liability of- 
fenses, and that does not necessarily reduce the criminal liability of other 
relevant entities. 


applica Bili Ty of general Defen SeS 
To ai criminal |ia Bili Ty 


Can an al system be insane? An infant? Intoxicated? Caught in situations 
of self-defense, necessity, or duress? One of the most common uses of 
Al technology is for guarding, as noted in chapter 1.1 For example, the 
South Korean government uses al robots as soldiers along its border with 
North Korea, and since 2012 as prison guards.” This technology faces par- 
ticular threats to the software, aimed at weakening its ability to discharge 
its guarding functions. This technology may also face dilemmas of the type 
humans encounter in everyday life. If a prison guard apprehends an escap- 
ing prisoner and the only way to stop him is by causing injury, is it right 
for the robot to injure a human for that purpose? 

Consider a case in which hackers manage to break into the main 
computer controlling robots that function as prison guards. The hackers’ 
purpose is to help a friend escape from prison by incapacitating some 
of the robots. The hackers plant a virus in the system, which immedi- 
ately affects the robots, preventing them from forming the correct image 
of the factual reality, and causes them to ignore some of the rules of the 
system. Some of the prisoners use this situation to attempt an escape. 
One prisoner captures a human guard and takes his uniform, trying to 
escape by wearing it. This prisoner is caught by a human guard, how- 
ever, and a struggle ensues. A robot witnessing the situation misinter- 
prets it as a struggle between two prisoners, and approaches the two men 
with the intention of intervening to stop the violence. When the human 
guard orders the robot to keep away, the robot interprets this as a threat 
to its mission, and grabbing the guard with its hydraulic arm, the robot 
smashes him against the wall. The disguised prisoner escapes, and the 
guard dies. The legal question is, who is to be held criminally liable for 
the guard’s death? 

The software controlling these robots is considered strong AT technol- 
ogy, so the robots are capable of consolidating awareness and mens rea in 
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the context of criminal law. Analysis of the records of these robots reveals 
that the mens rea reguirements of homicide are met in full. But this is not 
the regular course of conduct for these robots, which are programmed not 
to be violent unless it is absolutely necessary. Accurate analysis of this 
case requires the application of the general defenses of criminal law to 
Al technology. This is the question that this chapter seeks to answer: are 
general defenses applicable to ar criminal liability? 


9.1. The funcTion of general defenses in cri Minal laW 


General defenses in criminal law are complementary to the mental ele- 
ment requirement; both address the offender’s fault in the commission 
of the offense. The mental element requirement represents the positive 
aspect of the fault (what should be present in the offender’s mind during 
the commission of the offense), and the general defenses are the nega- 
tive aspect of the fault (what should be missing from the offender’s mind 
during the commission of the offense). For example, awareness is part 
of the mental element requirement (mens rea), and insanity is a general 
defense. Thus, in mens rea offenses, the offender must be aware and must 
not be insane. 

Therefore, the fault requirement in criminal law consists of the mental 
element requirement and the general defenses. The general defenses were 
developed in the ancient world to prevent injustice in certain types of 
cases. For example, someone who killed another out of self-defense was 
not criminally liable for the homicide, because the person lacked the re- 
quired fault to cause death. An authentic factual mistake by the offender 
regarding the intentional commission of an offense was considered as ne- 
gating the required fault for imposing criminal liability.* In the modern era, 
the general defenses became wider and more conclusive, but the common 
factor of all general defenses remained the same. 

All general defenses in criminal law are part of the negative aspect of 
the fault requirement, as they are intended to negate the offender’s fault. 
The deep abstract question behind the general defenses is whether the 
commission of the offense was not imposed on the offender in some way. 
For example, when a person really acts in self-defense, the offense is con- 
sidered to be imposed on her. To save her own life, which is considered a 
legitimate purpose, the offender had no choice but to act in self-defense. 
Naturally, she could have chosen to give up her life, but that is not con- 
sidered to be a legitimate requirement because it goes against the natural 
instinct of every living creature. 
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General defenses can be divided into two main types: exemptions and 
justifications.” Exemptions are general defenses related to the personal 
characteristics of the offender (in personam), and justifications are related 
to the characteristics of the factual event (in rem). The personal charac- 
teristics of the offender may negate the fault for the commission of the 
offense, regardless of the factual characteristics of the event or the exact 
identity of the offense. In exemptions, the personal characteristics of the 
offender are sufficient to prevent imposition of criminal liability for any 
offense. 

For example, a child under the age of legal maturity is not criminally 
liable for any offense factually committed by him. The same is true for in- 
dividuals who committed the offense at a time when they were considered 
to be insane. The exact nature of the offense committed by the individual 
is immaterial for the imposition of criminal liability. It may be relevant for 
subsequent steps toward the treatment or rehabilitation of the offender, but 
not for the imposition of criminal liability. 

Justifications are impersonal general defenses, and as such they do not 
depend on the identity of the offender, but only on the factual event that 
took place. The personal characteristics of the individual are immaterial 
for justifications. For example, one person is attacked by another person, 
and her life is in danger. She pushes the attacker away, which is consid- 
ered assault unless it is done with consent. In this case, the person doing 
the pushing claims self-defense regardless of her identity, the identity of 
the attacker, or any other of their personal attributes, because self-defense 
has to do with only the factual event itself. 

Because justifications are impersonal, they also have a prospective 
value. Not only is the individual not criminally liable if she acted with 
justification, but this is the way in which she should have been acting. Jus- 
tifications define not only types of general defense, but also proper behav- 
ior.é Therefore, individuals should defend themselves in situations that 
require self-defense, even if this may appear as an offense. This is not true 
for exemptions. A child below the maturity age is not supposed to commit 
offenses even if no criminal liability is imposed on him, and neither are 
insane individuals. 

The prospective behavioral value of justifications expresses the social 
values of society. If we accept self-defense, for example, as legitimate jus- 
tification, this means that we prefer people to protect themselves when 
the authorities are unable to protect them. We prefer to reduce the monop- 
oly of the state on power by legitimizing self-assistance rather than leave 
people vulnerable and helpless. We do not coerce individuals to act in 
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their self-defense, but if they do, then we do not consider them criminally 
liable for the offense committed through self-defense. 

Both exemptions and justifications are general defenses. The term gen- 
eral defenses relates to defenses that may be applied to any offense, and 
not to a certain group of offenses. For example, infancy can be applied to 
any offense as long as it was committed by an infant. By contrast, some 
specific defenses can be applied only to specific offenses or types of of- 
fenses. For example, in some countries in the case of statutory rape, a 
defense is applicable if the age gap between the defendant and the victim 
is less than three years. This defense is unique to statutory rape, and it is 
irrelevant for any other offense. Exemptions and justifications are classi- 
fied as general defenses. 

As defense arguments, general defenses are argued positively by the 
defense. If the defense chooses not to raise these arguments, they are not 
discussed in court, even if participants at the trial understand that such an 
argument may be relevant. It is not sufficient to state the general defense: 
its elements must be proven by the defendant. In some legal systems, the 
general defense is proven by raising a reasonable doubt about the presence 
of the elements of the defense. In other legal systems, the general defense 
must be proven by a preponderance of the evidence. Thus, the prosecution 
has the opportunity to refute the general defense. 

General defenses of the exemption type include infancy, loss of self- 
control, insanity, intoxication, factual mistakes, legal mistakes, and sub- 
stantive immunity. General defenses of the justification type include self- 
defense (including defense of dwelling), necessity, duress, superior orders, 
and de minimis defense. All these general defenses may invalidate the 
offender’s fault. The question is whether these general defenses are appli- 
cable to a1 technology in the context of criminal law. We answer this ques- 
tion now, discussing exemptions and justifications separately. 


9.2. exeMpTions 


Exemptions are general defenses related to the personal characteristics 
of the offender (in personam), as noted previously.’ The applicability of 
exemptions to the criminal liability of a1 entities raises the question of the 
capability of these entities to form the personal characteristics required 
for general defenses. For example, the question of whether an ai system 
could be insane is rephrased to ask whether it has the mental capability of 
forming the elements of insanity in criminal law. This question, mutatis 
mutandis, is relevant to all exemptions, as we will discuss. 


When robots Kill - 124 


5.2.1. infancy 

Could an ai system be considered an infant for the purposes of criminal 
liability? Since ancient times, children under a certain biological age were 
not considered criminally liable (doli incapax). The difference between 
the various legal systems was in the exact age of maturity. For example, 
Roman law set the age of maturity at seven years. This defense is deter- 
mined by legislation? and case law.” It is clear that the cutoff age is biologi- 
cal and not mental, primarily for evidentiary reasons, because biological 
age is much easier to prove." It was presumed, however, that the biological 
age matches the mental age. 

If the child’s biological age exceeds the lower threshold but is under the 
age of full maturity (for example, fourteen in the United States), the mental 
age of the child is examined based on evidence (for example, expert testi- 
mony).!? The conclusive examination is whether the child understands his 
own behavior and its wrong character," and whether he understands that 
he may be criminally liable for the offense, as if he were mature. But there 
may still be some procedural changes in the criminal process compared to 
the standard process (for example, juvenile court, the presence of parents, 
or more lenient punishments). 

The rationale behind this general defense is that children under a cer- 
tain age, whether biological or mental, are presumed to be incapable of 
forming the fault required for criminal liability.* The child is not inca- 
pable of mentally containing the fault and understanding the full social 
and individual meanings of criminal liability, so that imposition of crimi- 
nal liability would be irrelevant, unnecessary, and vicious. Consequently, 
children are not criminally liable but rather are educated, rehabilitated, 
and treated.** The question, in our case, is whether this rationale is rele- 
vant only to humans but to other legal entities as well. 

The general defense of infancy is not considered applicable to corpo- 
rations. There are no “child corporations,” so the moment a corporation 
is registered it exists, and criminal liability may be imposed on it legally; 
therefore, the rationale of this general defense is irrelevant for corpora- 
tions. A child lacks the mental capability to form the required fault be- 
cause human consciousness is underdeveloped at this age. As the child be- 
comes older, his or her mental capacity develops gradually until it reaches 
full understanding of right and wrong. At this point, criminal liability 
becomes relevant. 

The mental capacity of corporations does not depend on their chrono- 
logical “age,” that is, the date of registration, and it is considered to be 
constant. Moreover, the mental capacity of a corporation derives from its 
human officers, who are mature entities. Consequently, there is no legiti- 
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mate justification for the general defense of infancy to be applied to cor- 
porations. The question of interest to us is whether ar systems resemble 
humans or corporations in this context. 

The answer differs for different types of al systems, and we must dis- 
tinguish between fixed ai systems and dynamically developing ones. Fixed 
AI systems begin their activity with the capacities they are going to have 
throughout their life cycle. These systems do not experience any change 
in their capacities with the passage of time. Consequently, their capacity to 
form mental requirements (for example, awareness, intent, or negligence) 
must be examined at every point where an offense has been committed, 
and the general defense of infancy does not apply. 

But the starting and end points of dynamically developing al systems 
are different. Their capacities, including mental ones, develop over time 
through machine learning or other techniques. If the system began its ac- 
tivity without the mental capacities required for criminal liability, and at 
some point it developed such capacities, then the time between the start- 
ing point and the point when it possesses such capacities parallels that of 
childhood. During this period, the system does not have the capabilities 
required for the imposition of criminal liability. 

But if the mental capacity of the ar system is already developed at the 
time the offense is committed, the question arises whether the infancy 
defense is still relevant because if the system possesses the required capa- 
bilities, it is criminally liable regardless of the point in time when these 
capabilities became available. The answer to this question is similar to the 
rationale for the general defense for human offenders. We can examine 
the mental capacity of each child at any age and determine individually 
whether the child possesses the required capabilities, but this would be 
very inefficient. Any preschooler who may have hit another one in day 
care is immediately identified as not having the mental capacities required 
for criminal liability. 

The other way is to establish a legal presumption whereby children 
under a certain age are not criminally liable. This is the situation with 
Al systems. Massive production of strong al systems (for example, dozens 
of prison guards or military robots), having the exact same capabilities and 
using the same learning techniques, makes it unnecessary to evaluate the 
mental capacities of each individual robot. If we know empirically that a 
given system possesses all required mental capabilities after 264 hours of 
activity, and if such a robot commits an offense before having operated for 
264 hours, the robot is presumed to be in its childhood and no criminal 
liability is imposed. 

If the prosecution insists that the individual system possesses the re- 
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quired mental capacities already in its “infancy,” or if the defense insists 
that the system does not possess the required mental capacities despite 
the fact that it has passed its infancy, the actual mental capacities of the 
systems can be examined and evaluated specifically. This would not be 
substantively different from the natural gaps between the biological and 
mental ages of humans. When the argument is made that a seventeen- 
year-old human offender is mentally underdeveloped, the court examines 
the offender’s mental capacities and decides whether that person has the 
required capacity to become criminally liable. 

This rationale is relevant for both humans and dynamically developing 
AI systems, but not for corporations. Therefore, it seems that the general 
defense of infancy can be relevant for this type of al systems under the 
right circumstances. 


5.2.2. | oss of Self-c ontrol 
Could an ai system experience loss of self-control in the context of crimi- 
nal liability? Loss of self-control is a general defense that has to do with 
one’s inability to control one’s bodily movements. When the reason for this 
inability is mental disease, it is considered insanity,” and when it is the 
effect of intoxicating substances, it is considered intoxication.* But the 
general defense of loss of self-control is more general than insanity and in- 
toxication because it does not require a specific type of reason for the loss 
of self-control. Whenever the offender’s bodily movements are not under 
his or her full control, this defense may be used if its conditions are met. 

The general rationale of this defense is that uncontrollable bodily 
movement does not reflect the offender’s will, and therefore it should not 
be the basis for the imposition of criminal liability. Thus, the uncontrolla- 
ble reflexes of an individual may be an expression of loss of self-control.”° 
For example, when a physician, during a routine medical checkup, taps 
a patient’s knee, she triggers a reflex that causes the patient’s leg to move 
forward. The leg may end up kicking the physician, which ordinarily may 
have been considered assault, but because in this instance it is the result 
of a reflex, the general defense of loss of self-control is applicable, and no 
criminal liability is imposed. 

But for the general defense to be applicable, the loss of self-control 
must be total. In this example, if the patient tapped his own knee on pur- 
pose for the reflex to be triggered, and as a result his leg kicked the physi- 
cian, the general defense is not applicable because the assault reflects the 
offender’s will. Consequently, two accumulative conditions must be met 
for the loss of self-control defense to be applicable: (1) inability to control 
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the self-behavior, and (2) inability to control the conditions under which 
that behavior occurred. 

In our example, the patient did not control his self-behavior (the 
reflex), but he controlled the conditions for its occurrence (tapping the 
knee); therefore, the general defense of loss of self-control is not applicable 
to him. 

Many types of situations have been recognized as loss of self-control, 
including automatism (acting without aware central control over the 
body),™ convulsions, post-epileptic states,” post-stroke states,” organic 
brain diseases, diseases of the central nervous system, hypoglycemia, hy- 
perglycemia,“ somnambulism (sleepwalking),” extreme sleep depriva- 
tion,” side effects of bodily” or mental traumas,” blackout situations,” 
side effects of amnesia*® and brainwashing,*! and many more.*” The cause 
for the loss of self-control is immaterial for meeting the first condition. As 
long as the offender is indeed incapable of controlling his or her behavior, 
the first condition is fulfilled. 

The second condition refers to the cause for being in the first condi- 
tion. If the cause was controlled by the offender, he is not considered to 
have lost his self-control. Controlling the conditions for losing the self- 
control is controlling the behavior. Therefore, when the offender controls 
the conditions for moving in and out of control, he cannot be considered 
to have lost his self-control. In Europe, the second condition is based on 
the doctrine of actio libera in causa, which dictates that if the one controls 
one’s entry into a situation that is out of control, the general defense of loss 
of self-control is rejected.** 

Accordingly, A1 systems can experience loss of self-control in the con- 
text of criminal law due to external and internal causes. For example, a 
human pushes an al robot onto another human. The robot being pushed 
has no control over that movement. This is an example of an external 
cause for loss of self-control. If the pushed robot makes nonconsensual 
physical contact with the other person, this may be considered an assault. 
The mental element required for assault is awareness, so if the robot is 
aware of that physical contact, both factual and mental elements require- 
ments of the assault are met. 

If the ar robot were human, it would have probably claimed the gen- 
eral defense of loss of self-control. Thus, although both mental and fac- 
tual elements of the assault are fulfilled, no criminal liability is imposed 
because commission of the offense was involuntary, or due to loss of self- 
control. This general defense would prevent the imposition of criminal 
liability on human offenders, and it should also prevent the imposition 
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of criminal liability on robots. If the robot has no AT capabilities for con- 
solidating awareness, there is no need for this defense because the robot 
would be functioning as a mere tool. The ar robot, however, is aware of 
the assault. 

The ability of the ar robot to meet the mental element requirement of 
the offense makes it necessary to apply the general defense, which func- 
tions identically with humans and robots. An example of internal cause 
for loss of self-control is the case of an internal malfunction or technical 
failure that causes uncontrolled movements by the robot. The robot may 
be aware of the malfunction and still not be able to control it or correct it. 
This is also the case for the general defense of loss of self-control. Whether 
the cause for the loss of self-control is external or internal is relevant to the 
applicability of this general defense. 

If the robot controlled these causes, however, the defense is not ap- 
plicable. For example, if the robot physically caused a person to push it 
(the robot) onto another person (external cause), or if the robot caused the 
malfunction knowing the probable consequences of this for its mecha- 
nism (internal cause), then the second condition of the defense is not met 
and the defense is not applicable. The situation is the same for humans. 
It appears, therefore, that the general defense of loss of self-control is ap- 
plicable to al systems. 


5.2.3. insanity 

Can an al system be considered insane for the purposes of criminal liabil- 
ity? Insanity has been known to humanity since the fourth millennium 
BC,” but at that time it was considered to be a punishment for religious 
sins,” and therefore there was no need to look for cures.* Only since the 
middle of the eighteenth century has insanity been explored as a mental 
disease, and its legal aspects considered.” In the nineteenth century, the 
terms insanity and moral insanity described situations in which the indi- 
vidual lost his or her moral orientation, or had a defective moral compass, 
but was aware of common moral values.*® 

Insanity was diagnosed as such only in case of significant deviations 
from common behavior, especially sexual behavior.” Since the end of 
the nineteenth century, it has been understood that insanity is mental 
malfunctioning, and that at times it is not expressed in deviations from 
common behavior. This approach lies at the foundation of the understand- 
ings of insanity in criminal law and criminology.*® Mental diseases and 
defects were categorized according to their symptoms and respective med- 
ical treatments, and their effect on criminal liability was explored and re- 
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corded. But the different needs of psychiatry and criminal law produced 
different definitions for insanity. 

For example, the early English legal definition of an insane person 
(“idiot”) was not being able to count to twenty. In psychiatry, such a 
person is not considered insane. Criminal law needed a clear and con- 
clusive definition of insanity, whereas psychiatry had no such needs. In 
most modern legal systems, today’s legal definition of insanity is inspired 
by two nineteenth-century English tests. One is the M’Naughten rules of 
1843,” and the second is the irresistible impulse test of 1840.** The com- 
bination of these two tests ensures that the general defense of insanity 
complies with the structure of mens rea, as discussed previously. 

The legal definition of insanity has both cognitive and volitive aspects. 
The cognitive aspect of insanity addresses the ability to understand the 
criminality of the conduct, whereas the volitive aspect addresses the abil- 
ity to control the will. Thus, if a mental disease or defect causes cognitive 
malfunction (difficulty to understand the factual reality and the criminal- 
ity of the conduct) or volitive malfunction (irresistible impulse), the condi- 
tion is considered insanity from the legal point of view. This is the con- 
clusive common test for insanity.“ It fits the structure of mens rea, which 
also contains both cognitive and volitive aspects, and it is complementary 
to the mens rea requirement.“ 

This definition of insanity is functional and not categorical. For an 
individual to be considered insane, it is not necessary to have a mental ill- 
ness that appears on a certain list of mental diseases. Any mental defect, 
of any type, can be the basis for insanity as long as it causes cognitive or 
volitive malfunctions. The malfunctions are examined functionally and 
not necessarily medically, and they need not appear on a list of mental 
diseases.“ Therefore, a person may be considered insane by criminal law 
and perfectly sane from the point of view of psychiatry, as in the case of 
a cognitive malfunction that is not categorized as a mental illness. The 
opposite situation, whereby a person is sane for the purposes of criminal 
law but insane psychiatrically, is also possible, as in the case of a mental 
disease that does not cause any cognitive or volitive malfunction. 

The insane person is presumed to be incapable of forming the fault re- 
quired for criminal liability. The question is whether the general defense 
of insanity is applicable to ar systems. The general defense of insanity re- 
quires a mental, or inner, defect that causes cognitive or volitive malfunc- 
tion. There is no need to demonstrate any specific type of mental disease, 
and any mental defect is satisfactory. The question is, how can we know 
about the existence of that “mental defect”? Because the mental defect is 
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examined functionally and not based on certain categories, the symptoms 
of that mental defect are critical for its identification. 

The inner defect causes cognitive or volitive malfunction, whether it 
is classified as a mental disorder, or a chemical imbalance in the brain, or 
an electric imbalance in the brain, and so on. The inner cause is exam- 
ined based on its functional effect on the human mind. This is the legal 
situation with humans, and the same holds true for A1 systems. The more 
complicated and advanced the AT system, the higher the probability of 
inner defects mostly in the software, but also in the hardware. Some inner 
defects do not cause the AI system to malfunction, but others do. If the 
inner defect causes a cognitive or volitive malfunction of the AT system, 
this matches the definition of insanity in criminal law. 

Because strong AI systems are capable of forming all mens rea com- 
ponents, as we have discussed, and given that these components consist 
of cognitive and volitive components owing to the mens rea structure, it 
is quite likely that some inner defects can cause these capabilities to mal- 
function. When an inner defect causes such a malfunction, this matches 
the definition of insanity in criminal law. Partial insanity is applicable 
when the cognitive or volitive malfunctions are not complete. Temporary 
insanity is applicable when these malfunctions affect the offender (human 
or AI system) for a certain period of time.*” 

It is possible to argue that this is not the typical character of the insane 
person because it does not match the concept of insanity reflected in psy- 
chiatry, culture, folklore, literature, and even the movies. Nevertheless, 
it is insanity from the perspective of criminal law. First, the criminal law 
definition of insanity differs from its definitions in psychiatry, culture, and 
so on, and it is the definition that is used for humans. There is no reason 
that a different definition should be used for a1 systems. Second, criminal 
law does not require a mental disease for human insanity, so why should 
we require it for A1 systems? 

The definition of insanity used by criminal law may seem too techni- 
cal, but if the offender meets its requirements, it is applied. If both human 
and al offenders are capable of meeting the insanity requirement in crimi- 
nal law, there is no legitimate reason for making the general defense of 
insanity applicable for one but not the other. Consequently, it appears that 
the general defense of insanity is applicable to AT systems. 


5.2.4. intoxication 
Can an A1 system be considered intoxicated for the purposes of criminal li- 
ability? The effects of intoxicating substances have been known to human- 
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ity since prehistory. In the early law of ancient times, the term intoxication 
referred to drunkenness as a result of ingesting alcohol. Later, when the 
intoxicating effects of other materials became known, the term was ex- 
panded." Until the beginning of the nineteenth century, intoxication was 
not accepted as a general defense. The Archbishop of Canterbury wrote 
in the seventh century that imposing criminal liability on a drunk person 
who committed homicide was justified for two reasons: first, the drunken- 
ness itself, and second, the homicide of a Christian person.” 

Drunkenness was conceptualized as a religious and moral sin, and 
therefore it was considered immoral to let offenders be exempt from crim- 
inal liability for being drunk." Only in the nineteenth century did the 
courts undertake a serious legal discussion of intoxication, made possible 
by legal and scientific developments that have produced the understand- 
ing that an intoxicated person is not necessarily mentally competent for 
purposes of criminal liability (non compos mentis). 

From the beginning of the legal evaluation of intoxication in the nine- 
teenth century, the courts distinguished between cases of voluntary and 
involuntary intoxication.** Voluntary intoxication was considered to be 
the offender’s fault, so it could not be the basis for exemption from crimi- 
nal liability. Nevertheless, voluntary intoxication could be considered as a 
relevant circumstance for the imposition of a more lenient punishment.” 
Moreover, voluntary intoxication could be used to refute premeditation in 
first-degree murder cases.5* Courts have distinguished between cases based 
on the reasons for becoming intoxicated. Voluntary intoxication grounded 
in the will to commit an offense was considered different from voluntary 
intoxication undertaken for no criminal reason.” 

By contrast, involuntary intoxication was recognized and accepted as 
an exemption from criminal liability." Involuntary intoxication is a situa- 
tion imposed on the individual, so it is not just and fair to impose criminal 
liability in such situations. Thus, the general defense of intoxication has 
two main functions. When intoxication is involuntary, the general defense 
prevents imposition of criminal liability. When it is voluntary but not in- 
tended for the commission of an offense, intoxication is a consideration 
for the imposition of a more lenient punishment. 

The modern understanding of intoxication includes any mental effect 
caused by an external substance (for example, chemicals). The required 
mental effect matches the structure of mens rea, discussed earlier. Conse- 
quently, the effect may be cognitive or volitive.5” The intoxicating effect 
may relate to the offender’s perception, understanding of factual reality, 
or awareness (cognitive effect); or it may affect the offender’s will, includ- 


When robots Kill - 132 


ing the creation of an irresistible impulse (volitive effect). Intoxication is 
caused by an external substance, but there is no closed list of substances 
that may be illegal (heroin, cocaine, and so on) or perfectly legal (alcohol, 
sugar, pure water, and so on). 

The effect of the external substance on the individual is subjective, 
and various people may be affected differently by the same substances in 
the same quantities. Sugar may produce hyperglycemia in one person, but 
another person is barely affected by it. Pure water may result in an imbal- 
ance of the electrolytes in one person, but another is barely affected by it. 
Cases of addiction raised the question whether the absence of the external 
substance may also be considered a cause for intoxication. For example, 
when a person addicted to narcotics is in withdrawal, he or she may expe- 
rience cognitive and volitive malfunctions as a result of the drug’s absence. 

Consequently, the cognitive and volitive effects of narcotics addiction 
are considered intoxication by criminal law.® To determine the question of 
voluntary or involuntary intoxication (in case the addicted person wanted 
to begin the process of weaning), it is the cause for the addiction, not for 
the weaning, that is examined to be voluntary or involuntary. Thus, in- 
toxication is examined through a functional evaluation of its cognitive and 
volitive effects on the individual, regardless of the exact identity of the ex- 
ternal substance that is the cause of these effects. The question is whether 
the general defense of intoxication is applicable to al systems. 

As already noted, the general defense of intoxication requires an exter- 
nal substance (for example, the presence or absence of a certain chemical) 
that has cognitive or volitive effects on the inner process of conscious- 
ness. For example, the manufacturer of ar robots wanted to reduce produc- 
tion costs, and used inexpensive materials. After a few months of opera- 
tion, a process of corrosion began in some of the initial components of the 
robots, and as a result, transfer of information was deficient and affected 
the awareness process. Technically, this is similar to the effect of alcohol 
on human neurons. 

In another example, a military A1 robot was activated in a civilian area 
after a real or simulated chemical attack. As a result of exposure to the 
poisonous gas, parts of the robot’s hardware were damaged, and the robot 
began to malfunction, became unable to properly identify people, and 
started to attack innocent civilians. Subsequent analysis of the robot's re- 
cords showed that exposure to the gas was the only reason for the attacks. 
If the robot had been human, the court would have accepted the general 
defense of intoxication and acquitted the defendant. Should not the same 
procedure also apply to al systems? 
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If there is no difference between the effects of external substances on 
humans and ar systems from a functional point of view, then there is no 
justification for applying the general defense of intoxication to one and 
not the other. Strong ar systems can possess both cognitive and volitive 
inner processes, as noted previously.” These processes can be affected by 
various factors, and when they are affected by external substances, as in 
the examples just presented, the requirements of intoxication as a general 
defense are met. Thus, if exposure to certain substances affects the cogni- 
tive and volitive processes of an al system in a way that causes it to commit 
an offense, there is no reason that intoxication as a general defense should 
not be applicable. 

It may be true that ar systems cannot become drunk on alcohol or have 
delusions following the ingestion of drugs, but these effects are not the 
only possible ones related to intoxication. If a human soldier attacks his 
friends as a result of exposure to a chemical attack, his argument for in- 
toxication is accepted. If this exposure has the same substantive and func- 
tional effect on both humans and ai systems, there is no legitimate reason 
for making the general defense of intoxication applicable to one type of 
offender but not another. Consequently, it appears that the general defense 
of intoxication can be applicable to ai systems. 


5.2.5. factual mistake 
We are such stuff 
As dreams are made on, and our little life 
Is rounded with a sleep. 
—William Shakespeare, The Tempest, 1611 


Can an AI system be considered factually mistaken for the purposes of 
criminal liability? The general defense of factual mistake provides a re- 
vised perspective of the cognitive aspect of consciousness. In our discus- 
sion of awareness, the assumption was that there is a factual reality of 
which the individual may or may not be aware.™ This reality was consid- 
ered constant, objective, and external to the individual. But the only way 
an individual can know about the factual reality is through the process 
of awareness, that is, perception by senses and understanding of the fac- 
tual data. 

Only when our brain tells us that this is the factual reality do we be- 
lieve it; we have no other way to evaluate reality’s existence. But sights, 
sounds, smells, or physical pressure may be simulated, and the human 
brain may be stimulated to feel them, although they may not necessarily 
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exist. The ultimate example is dreaming. Often, we dream without know- 
ing that we are dreaming, and we cannot distinguish between dream and 
factual reality while we are dreaming. In our dreams we see sights, hear 
sounds, smell, feel, talk, or run as if we were facing factual reality. 

What makes the dream different from factual reality in the eyes of 
the dreamer? For most people, the dream is a dream simply because the 
dreamer eventually exits it by waking up. But what would happen if, on 
the contrary, we were to wake up in a dream? This option is not accepted 
by most people because our intuition tells us that if we open our eyes, if 
we are in bed, and if it is morning, the dream portion of this day has ended 
and the waking portion begins. But when we are in the midst of the dream, 
we have no reliable way of distinguishing dreams from what we call “fac- 
tual reality” (within quotation marks, because we do not have a reliable 
way of verifying it). 

We “see” things with our brain, not with our eyes. Our eyes are merely 
light sensors. The light is converted into electrical currents and chemi- 
cal processes that carry messages between neurons and form the “sight” 
by stimulating the brain in the right spots. But it is possible to imitate 
this stimulation. Electrodes can be attached to the brain to stimulate it in 
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the appropriate regions and make it “see,” “hear,” or “feel.” The stimu- 
lated brain becomes aware of these images, sounds, and so on, and people 
would consider them factual reality. 

If we begin to doubt our awareness of the factual reality, we must also 
consider the problem of perspective. Even if we assume that we experi- 
ence the factual reality (without guotation marks), we can experience it 
only through our subjective perspective, which is not necessarily the only 
perspective of that reality. For example, if we cut out a triangular shape 
from a piece of cardboard, we may perceive a triangle if we view it facing 
its flat surface; but if we rotate it by 90 degrees, we perceive it as a straight 
line. The issue of perspective can be crucial if we associate different inter- 
pretations with the different perspectives. 

For example, two people see a man holding a long knife and hear him 
telling a woman that he is about to kill her. One person understand this 
as a serious threat to the woman’s life, and calls police or attempts to in- 
tervene and save the woman; the other person understands this as part of 
a show (for example, a street performance) that requires no intervention. 
The deep question in criminal law in this context is, what should be the 
factual basis of criminality: the factual reality as it actually occurred, or 
what the offender believed to be the factual reality, although it may not 
have actually taken place? 
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For example, a man charged with rape admits that he and the com- 
plainant had sexual intercourse, but proves that he had really believed 
that it was carried out consensually; and the prosecution proves that the 
complainant did not consent. The court believes both, and both are telling 
the truth. Should the court acquit or convict the defendant? Since the sev- 
enteenth century, modern criminal law has preferred the subjective per- 
spective of the individual (the defendant) about the factual reality to the 
factual reality itself. The general concept is that the individual cannot be 
criminally liable except for “facts” that he or she believed to be knowing, 
regardless of whether they actually occurred in factual reality. 

In most legal systems, the limitations imposed on this concept are evi- 
dentiary, so that the defendant’s argument would be considered true and 
authentic, if proven by the defendant. When the argument is considered 
true and authentic, this becomes the basis for the imposition of crimi- 
nal liability. In our example, the defendant is acquitted because he truly 
believed the intercourse was consensual. If the defendant’s perspective 
negates the mental element requirement, the factual mistake works as a 
general defense that leads to the defendant’s acquittal. 

The general defense of factual mistake is applicable not only in mens 
rea offenses, but also in negligence and strict liability offenses. The differ- 
ence between them is in the type of mistake required. In mens rea offenses, 
any authentic mistake negates awareness of the factual reality and is con- 
sidered adequate for that general defense. In negligence offenses, the mis- 
take is also required to be reasonable for the defendant to be considered 
to have acted reasonably. In strict liability offenses, the mistake must be 
inevitable despite the defendant having taken all reasonable measures to 
prevent it.® The question is whether the general defense of factual mistake 
is applicable to a1 systems. 

Both humans and AI systems can experience difficulties, errors, and 
malfunctions in the process of awareness of the factual reality. These dif- 
ficulties may occur both in the process of absorbing the factual data by the 
senses and in the process of creating the relevant general image about the 
data. In most cases, such malfunctions result in the formation of an inner 
factual image that differs from the factual reality as the court understands 
it. This is a factual mistake concerning the factual reality. Factual mistakes 
are part of our everyday life, and they are a common basis for our behavior. 

In some cases, factual mistakes by both humans and AT systems can 
lead to the commission of an offense. This means that according to factual 
reality the action is considered an offense, but not so according to the sub- 
jective inner factual image of the individual, which happens to involve a 
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factual mistake. For example, a human soldier mistakenly identifies his 
friend as an enemy and shoots him. For reasons unknown, the soldier 
who was shot wore the enemy’s uniform, spoke the enemy’s language, and 
looked as if he intended to attack the soldier who shot him. Although he 
was asked to identify himself, he did not comply with the request. In this 
case, the mistake is authentic, reasonable, and inevitable. 

If the human soldier claims a factual mistake, he will probably be 
acquitted (if indicted at all). If the soldier is not a human but a strong 
Al robot, should not the criminal law treat the robot soldier the same way 
that it treats the human soldier? The errors committed by the human and 
robot soldiers are substantively and functionally identical. The factual 
mistakes of both humans and ai entities have the same substantive and 
functional effects on cognition and on the perception of factual reality. As 
a result, there is no reason for preventing the use of the factual mistake as 
general defense for AT entities exactly as it is applied to humans. 

Computers do make mistakes. The probability of a mistake in math- 
ematical calculations by a computer may be low, but if the computer ab- 
sorbs mistaken factual data, the results of the calculations may be wrong. 
Calculations regarding required, possible, and impossible courses of action 
are affected by the data being absorbed, exactly as in the case of humans.” 
If factual mistakes have identical substantive and functional effects on 
humans and ai systems, there is no legitimate reason for making the gen- 
eral defense of factual mistake applicable to one type of offender and not 
to the other. Consequently, it appears that the general defense of factual 
mistake can be applied to AT systems. 


5.2.6. | egal mistake 

Could an ar system be considered legally mistaken for the purpose of crim- 
inal liability? A legal mistake is a situation in which the offender either 
misinterprets the law or is ignorant of it. The general idea behind this de- 
fense is that a person who does not know about a certain prohibition, and 
consequently commits an offense, does not consolidate the required fault 
for the imposition of criminal liability. Nevertheless, the mental element 
of the offenses does not include knowing about the prohibition. 

For example, the offense of rape requires a mental element of mens rea, 
which includes awareness of the commission of sexual intercourse with a 
woman and of the absence of consent. That offense does not require that 
the rapist know that rape is prohibited as a criminal offense. The mental 
element requirement of rape is satisfied by awareness of the factual ele- 
ment components, regardless of the rapist’s knowledge that rape is pro- 
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hibited. This is the legal situation for most offenses owing to prospective 
considerations of everyday life in society.” 

If offenders were reguired to know about the prohibition as a condi- 
tion for the imposition of criminal liability, they would be encouraged not 
to learn the law, and as long as they were ignorant of the law, they would 
enjoy immunity from criminal liability. If no such condition is reguired, 
the public is encouraged to inguire about the law, to know it, and to obey 
it. These prospective considerations do not fully match the fault reguire- 
ment in criminal law. As a result, criminal law seeks a balance between 
justice, the fault reguirement, and prospective considerations concerned 
with everyday life in society. 

Initially, considerations were entirely prospective. Roman law stated 
that ignorance of the law does not excuse the commission of offenses (ig- 
norantia juris non excusat), and until the nineteenth century, this was 
the general approach followed by most legal systems.” In the nineteenth 
century, when the culpability requirement in criminal law underwent a 
dramatic development, it became necessary to establish a balance, and the 
legal mistake was required to be made in good faith (bona fide)” and to 
reflect the highest state of mental element, that is, strict liability. Accord- 
ing to this standard, the legal mistake is an inevitable one, despite the fact 
that all reasonable measures have been taken to prevent it.” 

This high standard for legal mistakes is required for all types of of- 
fenses, regardless of their mental element requirement. Therefore, the gen- 
eral standard for legal mistakes is higher than that for factual mistakes. 
In this context, the main question in courts is whether the offender has 
indeed taken all reasonable measures to prevent the legal mistake, includ- 
ing reasonable reliance on statutes, judicial decisions,” official interpreta- 
tions of the law (including pre-rulings),” and the advice of private coun- 
sel.Z The question for us is whether the general defense of legal mistake is 
applicable to AT systems. 

Technically, if the relevant entity, whether a human or an AT system, 
is capable of fulfilling the mental element requirement of strict liability 
offenses, the entity also is capable of claiming legal mistake as a general 
defense. Because strong a1 systems have the ability to meet the mental el- 
ement requirement of strict liability offenses, as discussed earlier,” they 
also have the capabilities needed to make the general defense of legal mis- 
take applicable to them. The absence of the legal knowledge of a given 
issue can be proven by examining the records attesting to the knowledge 
of the AT entity, thereby meeting the good faith requirement as well. 

The fundamental meaning of the applicability of the legal mistake 
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defense to al systems is that the system was not restricted by any formal 
legal restriction, and it acted accordingly. If the Ar system contains a soft- 
ware mechanism that searches for such restrictions, and although the 
mechanism has been activated no such legal restriction has been found, 
then the general defense applies. Note, however, that the system’s ex- 
emption from criminal liability does not function as an exemption from 
criminal liability for the programmers or users of the system. If these 
persons could have restricted the activities of the system to strictly legal 
ones but failed to do so, they may be criminally liable for the offense 
based on perpetration-through-another or on probable consequence, as 
we have discussed. 

For example, an al system absorbs factual data about certain people, 
and it is required to analyze their personalities based on the data and pub- 
lish the results within certain guidelines. In one instance, the publication 
is considered criminal libel. If the records of the system show that it has 
not been constrained by any restriction regarding libelous publications 
and it either did not have the mechanism to search for such restrictions 
or it did have the mechanism but found no such restrictions, then the 
system is not criminally liable for libel. The manufacturer, programmers, 
and users may be criminally liable for the libel, however, as perpetrators- 
through-another or through probable consequence liability. 

An Al system may have broad knowledge about many types of issues, 
but it may not contain legal knowledge on every legal matter. The system 
may be searching for legal restrictions if it is designed to do that, but may 
not necessarily find any; this then becomes a case for the general defense 
of legal mistake. If legal mistakes have the same substantive and functional 
effects on both humans and AI systems, there is no legitimate reason to 
make the general defense of legal mistake applicable to one type of of- 
fender but not to the other. Consequently, it appears that the general de- 
fense of legal mistake is applicable to A1 systems. 


5.2.7. Substantive immunity 
Could an ai system have substantive immunity in the context of criminal 
law? Certain types of people enjoy substantive immunity from criminal 
liability by virtue of their office (ex officio). The immunity is granted to 
these people ex ante so that they are not troubled with issues of criminal 
law related to their office. Society grants these immunities because the of- 
fices held by such people are regarded to be much more important than 
the probable criminal offenses they may commit in the course of discharg- 
ing their duties. This immunity is not absolute, and it relates to offenses 
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committed by these people as part of discharging their official duty and 
for its sake. 

For example, a firefighter on a mission to save a woman’s life is hoisted 
by a crane and breaks the window to enter her apartment. This act ap- 
parently meets the factual and mental element requirements of several 
offenses (for example, break-in, entering, and property damage), but to 
enable the firefighter to discharge his lifesaving duty, he is granted sub- 
stantive immunity from criminal liability for these offenses. If, however, 
the firefighter had entered the apartment and raped the woman, the im- 
munity would have been immaterial because rape is not part of his duties. 

This type of immunity is substantive as it nullifies the criminal liabil- 
ity of the person to whom it is extended, but it does not prevent indictment 
or other criminal proceedings. The law explicitly lists the types of people 
to whom substantive immunity is extended (firefighters, police officers, 
soldiers, and so on). The general defense applies to these people only if 
they have committed the offense as part of their official duty, in order to 
discharge that duty, and in good faith (bona fide), that is, not exploiting 
the immunity to deliberately commit other criminal offenses. The ques- 
tion is whether the general defense of substantive immunity is applicable 
to AI systems. 

The firefighter in our example was human, and if indicted for causing 
property damage would claim substantive immunity. The court would 
most likely accept the claim and acquit him immediately. Let us assume, 
however, that the fire posed too heavy a risk for a human firefighter, and 
an Al robot was dispatched to save the woman’s life. The robot is equipped 
with a strong AI system and capable of making the same decision that 
a human firefighter would make under the circumstances. Is there any 
reason such a robot should not be granted similar immunity to that en- 
joyed by the human firefighter? 

If all the conditions for granting this general defense are met, there is 
no reason for using different standards toward humans and A1 systems. AI 
robots are already being used for official duties (for example, as guards), 
and inevitably at times they must commit offenses in the course of dis- 
charging their duties. For example, prison guards may physically assault 
escaping prisoners to prevent them from escaping. If these situations have 
the same substantive and functional effects on both humans and al sys- 
tems, then there is no legitimate reason for make the general defense of 
substantive immunity applicable to one type of offender but not to the 
other. Consequently, it appears that the general defense of substantive im- 
munity is applicable to AT systems. 
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9.3. JusTifica Tions 


Justifications are general defenses having to do with the characteristics of 
the factual event (in rem), as previously noted.” The applicability of justi- 
fications to the criminal liability of A1 systems raises the question of the ca- 
pability of these systems to be involved in such situations as self-defense, 
necessity, or duress. Is it legitimate to allow a robot to defend itself from 
attack? And if the attack is conducted by humans, is it legitimate to allow 
a robot to attack humans to ensure the robot’s safety? 

In general, because justifications are in rem general defenses, the per- 
sonal characteristics of the individual (human or al system) should be con- 
sidered immaterial. But these general defenses were designed for humans, 
taking into account human weaknesses and allowing for them. For ex- 
ample, self-defense was designed to protect the human instinct to protect 
one’s life. Is this instinct relevant to al systems, which are machines? If 
not, why would self-defense apply to machines? We will now explore the 
applicability of justifications as general defenses for AT systems. 


5.3.1. Self-Defense 

Self-defense is one of the most ancient defenses in human culture. Its 
basic function is to mitigate the society’s absolute monopoly on power 
according to which only society (the state) has the authority to use force 
against individuals. When one individual has a dispute with another, he 
or she is not allowed to use force, but must apply to the state (through the 
courts, police, and so on) for the state to solve the problem and use force 
if necessary. According to this concept, no power is left in the hands of 
individuals. 

For this concept to be effective, however, state representatives must be 
present at all times in all places. If one individual is attacked by another 
in a dark alley, she cannot retaliate, but must wait for the state represen- 
tatives, who may be unavailable at that time. To enable people to protect 
themselves from attackers in situations like this, society must partially re- 
treat from this concept. One such retreat is the acceptance of self-defense 
as a general defense in criminal law. Self-defense allows individuals to 
protect some values using force, skirting the monopoly on power enjoyed 
by the state. 

Being in a situation that warrants self-defense is considered to nul- 
lify the individual’s fault required for the imposition of criminal liability. 
This concept has been accepted by legal systems worldwide since ancient 
times, and eventually the defense has become wider and more accurate." 
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Its modern form allows individuals to repel attacks on legitimate interests. 
There are several conditions for this general defense to be applicable: 


1. The protected interest must be legitimate. Legitimate interests 
are the life, freedom, body, and propertys* of the individual or 
of other individuals. No previous introduction between the 
person threatened and the defender is reguired,#7 so that self- 
defense is not only of the “self.” 

2. The protected interest must be attacked illegitimately.** When a 
police officer, carrying a legal warrant, attacks an individual to 
make an arrest, the attack is considered legitimate. 

3. The protected interest must be in clear and imminent danger.” 

4. The act of self-defense must be carried out to repel the attack, 
and it must be proportional to it,” and immediate.” 

5. The defender must not be in a situation to control the attack or 
the conditions for its occurrence (actio libera in causa). 


If all these conditions are met, the individual is considered to be acting in 
self-defense, and therefore no criminal liability is imposed on him or her 
for the commission of the offense. But not every time an attack is repelled 
can it be considered self-defense; only when the act meets the listed con- 
ditions in full. The question is whether the general defense of self-defense 
is applicable to ar systems. The answer depends on the capabilities of the 
AI systems to meet the conditions just outlined, in the relevant situations. 

At the heart of self-defense is the legitimate protected interest. When 
the protected interest is that of another person (a human interest pro- 
tected by the ai system using self-defense), there seems to be no difficulty. 
Indeed, this is the broad basis for the legal activity of guard robots (of 
humans, prisoners, borders, dwellings, and so on). If these robots have the 
authority to repel attacks on human interests, self-defense is a legal justi- 
fication. But the question is whether the self-interest of an al system may 
also be legally protected by self-defense. Indeed, there are two questions 
here: one is moral, the other legal. 

Morally, the question is whether we accept the idea of a robot pro- 
tecting itself and having derivative rights, some of them constitutional.” 
The moral question has nothing to do with the legal one. Since the 1950s, 
the human approach to this moral question has generally been positive. 
Recall Asimov’s third law, which states, “A robot must protect its own 
existence, as long as such protection does not conflict with the First or 
Second Laws.” According to this law, under the right circumstances the 
robot is not only authorized to protect itself, but it must do so. As already 
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noted, however, this is not part of the legal question, and as far as the ap- 
plicability of self-defense to robots is concerned, only the legal question 
is relevant. 

The legal question is much simpler, and it asks whether an ar system 
has the ability to protect its own “life,” freedom, body, and property. To 
protect these legitimate interests, the AT system must first possess them. 
Only if an ar system possesses property can it protect it; otherwise it can 
protect only the property of others. The same is true concerning life, free- 
dom, and body. Currently, criminal law protects human life, freedom, 
and body. The question of whether a robot has analogous life, freedom, 
and body is one of legal interpretation, in addition to the moral questions 
involved. 

Criminal law recognized decades ago that the corporation, which is not 
a human entity, possesses life, freedom, body, and property. We return 
to this analogy in the next chapter when we discuss the punishments of 
AI systems, including capital punishment and imprisonment. But if the 
legal question concerning corporations, which are abstract creatures, has 
been decided affirmatively, it would be unreasonable to decide otherwise 
in the case of al systems, which physically simulate these human attri- 
butes much better than do abstract corporations. 

For example, in a prisoner escape, a prison-guard robot is attacked by 
the escaping prisoners, who intend to tie it up in order to incapacitate it 
and prevent it from interfering with their escape. Is it legitimate for the 
robot to defend its mission and protect its freedom? If the prisoners intend 
to break its arms, is it legitimate for the robot to defend its mission and 
protect its “body”? If the answers are analogous to those supplied for cor- 
porations, they are affirmative. In any case, even if the idea of an AT robot’s 
life, body, and freedom is not acceptable, despite the analogy with corpo- 
rations and despite the positive moral attitude, A1 systems still meet the 
first condition of self-defense by protecting the life, body, freedom, and 
property of humans. 

The other two legal conditions of self-defense do not appear to differ 
for humans and AI systems. The condition requiring an illegitimate attack 
depends on the attacker, not on the identity of defender, whether human or 
not. The condition of clear and imminent danger to the protected interest 
does not depend on the defender either. The nature of the danger is deter- 
mined by the attacker, not by the defender, and therefore this condition is 
met not through the behavior of the defender, but through the analysis of 
the attack, independent of the defender’s identity. 

The condition of repelling the attack by a proportional and immediate 
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action raises another question: is it legitimate for a robot to attack a human, 
even if the attack is in order to repel, by a proportional and immediate re- 
action, a human attack? This question raises two subsidiary questions. The 
first involves the legitimacy of preferring the defender’s rights over those 
of the attacker, regardless of their identity as humans, corporations, or 
Al systems. The second concerns restriction on AI systems as defenders 
that differ from the restrictions imposed on humans and corporations. 
Both subsidiary questions have legal and moral aspects. 

The first subsidiary question is generally answered with reference to 
risk taking by the attacker. Society creates peaceful social mechanisms 
for dispute resolutions, such as legal proceedings, arbitration, mediation, 
and so on. The attacker chooses not to use these peaceful mechanisms, but 
rather chooses to resort to illegal violence. When acting in this manner and 
bypassing the legal mechanisms of dispute resolution, the attacker takes 
the risk of triggering a response against the illegal attack. In this case, crim- 
inal law prefers the innocent reaction over the illegal action.” This answer 
recognizes no difference between various types of attackers and defenders. 

There may be several moral answers to the second subsidiary question, 
but none is relevant to the legal answer. Therefore, despite Asimov’s first 
law, noted earlier,** which prohibits robots from harming humans, robots 
are actually used in several ways that harm humans. Using robots in mili- 
tary and police functions (as soldiers, guards, armed drones, and so on)*? 
inherently implies the possibility of causing harm to humans. As long as 
this use has not been explicitly prohibited by criminal law, it is legitimate. 
It appears that legal systems worldwide have not only made their legal 
choice in this matter, but have made their moral choice as well, rejecting 
Asimov’s first law, which has been regarded as too panicky. 

Consequently, criminal law sees no legal problem with the possibil- 
ity that a robot may protect legitimate interests under illegal attack by 
humans. This question would not have been raised at all if the attacker 
were a robot, in which case the answer would be that the attack must be 
repelled with proportional and immediate action, whether the defender is 
human or not. The reaction is evaluated identically, regardless of whether 
the defender is human. Repelling means that the act is a reaction to the 
attack, which is the cause for the reaction. Immediate means that the reac- 
tion is subsequent to the attack. The reaction is not illegal revenge, but an 
act intended to neutralize the threat. 

Proportional means that the defender does not use excessive force to 
neutralize the threat. The defender must evaluate the threat and the means 
that may be used to neutralize it, and then choose the means that are not 
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excessive in relation to the specific threat under the given circumstances. 
Proportionality resembles reasonability in many ways, and some under- 
stand it as being part of reasonability. Therefore, to meet this condition, the 
Al system must have the capability of reasonableness, which is required for 
negligence as well, as discussed previously in that context.19 

Finally, it is required that the defender have no control over the attack 
or the conditions for its occurrence. This requirement is intended to 
impose criminal liability on persons who brought the attack upon them- 
selves and then attempt to save themselves from criminal liability by using 
the self-defense argument. The court is likely to seek out the deep reasons 
for the attack. If the defender was innocent in this context, the defense is 
applicable. This requirement is not applied differently to humans, corpo- 
rations, or AT entities. As long as the AT entity was not part of such a plot, 
the general defense of self-defense is applicable to it. The proof may be 
based on the ai system’s records. Consequently, it appears that the general 
defense of self-defense is applicable to Ar systems. 


5.3.2. necessity 

Can an al system be considered to be acting out of necessity in the criminal 
law context? Necessity is a justification of the same type as self-defense. 
Both are designed to mitigate the society’s absolute monopoly on power, 
discussed earlier. The principal difference between self-defense and ne- 
cessity is in the identity of the object of the response. In the case of self- 
defense, the defender’s reaction is against the attacker, whereas in the case 
of necessity it is against an innocent object (innocent person, property, 
and so on). The innocent object is not necessarily related to the cause of 
the reaction. 

For example, two people are sailing on the high seas. The boat hits an 
iceberg and sinks. Both people survive on an improvised raft, but have 
no water or food. After a few days, one of them eats the other in order to 
survive.’ The victim did not attack the eater and was not to blame for the 
crash; she was completely innocent. The attacker was also innocent, but 
he knew that if he did not eat the other person, he would definitely die. 
If the eater is indicted for murder, he can claim necessity (self-defense is 
not relevant in this case because the victim did not threaten the attacker). 

The traditional approach toward necessity is that under the right cir- 
cumstances, it can justify the commission of offenses (quod necessitas non 
habet legem). The traditional justification of the defense is the under- 
standing in criminal law of the weakness of human nature. The individual 
who acts under necessity is considered to be choosing the lesser of two 
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evils from his or her own point of view.’ In our example, if the attacker 
chooses not to eat the other person, they both perish; if he eats the other 
person, only one of them dies. Both situations are evil, but the lesser evil 
of the two is the one in which one person survives. The victim of neces- 
sity is not blamed for anything but is innocent, and the act is considered 
to be justified. 

Given that the act performed out of necessity is the individual’s, car- 
ried out when the authorities are not available, the general defense of ne- 
cessity partially reduces the applicability of the general concept whereby 
only society (the state) has a monopoly on power, as we have discussed. 
Being in a situation that requires an act of necessity is considered to nul- 
lify the individual’s fault required for the imposition of criminal liabil- 
ity. This concept has been accepted by legal systems worldwide since 
ancient times, and eventually the defense became broader and more ac- 
curate, so that its modern form allows individuals to protect legitimate 
interests.'°° 

There are several conditions for this general defense to be applicable: 


1. The protected interest must be legitimate. Legitimate interests 
are the life, freedom, body, and property of the individual or 
of other individuals. No previous introduction between the 
person threatened and the defender is required.” 

2. The protected interest must be in clear and imminent danger. 

3. The act of necessity must be directed against an external and 
innocent interest. 

4. The act of necessity must be carried out to neutralize the danger, 
and it must be proportional to it,“° and immediate. 

5. The defender must not be in a situation to control the danger or 
the conditions for its occurrence (actio libera in causa). 


If all these conditions are met, the individual is considered to be acting 
under necessity, and therefore no criminal liability is imposed on him or 
her for the commission of the offense. But not every danger neutralized 
by causing harm to an innocent interest can be considered necessity; only 
when the act meets the listed conditions in full. The question is whether 
the general defense of necessity is applicable to AT systems. The answer 
to this question depends on the capabilities of the a1 systems to meet the 
conditions just outlined, in the relevant situations. 

Four of the listed conditions—numbers 1, 2, 4, and 5—are identical 
with those of self-defense, mutatis mutandis; the difference is that instead 
of an attack on a legitimate interest, we are facing danger to that same in- 
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terest. The main difference between self-defense and necessity lies in one 
of the conditions. Whereas in self-defense the act is directed toward the 
attacker, in the case of necessity the act is directed toward an external or 
innocent interest. In the case of necessity, the defender must choose be- 
tween the lesser of two evils, one of which harms an external interest, who 
may be an innocent person having nothing to do with the danger. 

The question regarding AI systems in this context is whether they are 
capable of choosing the lesser of two evils. For example, an Ar-based lo- 
comotive transporting twenty passengers arrives at a junction of two rail- 
roads. A child is playing on one of the rails, but the second rail leads 
to a cliff. If the system chooses the first rail, the child will certainly die 
but the twenty passengers will survive. If it chooses the second rail, the 
child will survive, but given the vehicle’s speed and the distance from 
the cliff, it will certainly fall off the cliff and none of the passengers will 
survive. 

If the vehicle had been driven by a human driver who chose the first 
rail and killed the child, no criminal liability would be imposed on him 
owing to the general defense of necessity. An al system can calculate the 
probabilities for each option and choose the possibility with the lowest 
number of casualties. Strong al systems are already used to predict compli- 
cated events (for example, weather patterns), and calculating the probabil- 
ities in the locomotive example is considered much easier. The al system 
analyzing the possibilities is faced with the same two possibilities that the 
human driver has to consider. 

If the al system takes into account the number of probable casualties, 
based on its programming or as a result of the relevant machine learning, 
it will probably choose to sacrifice the child. This choice by the ar system 
meets all the conditions of necessity, so that if it were human, no criminal 
liability would be imposed on it owing to the general defense of neces- 
sity. Should the a1 system be treated differently, then? Moreover, if the AT 
system chooses the alternative option, and causes not the lesser but the 
greater of the two evils to occur, we would probably want to impose crimi- 
nal liability (on the programmer, the user, or the AT system) in exactly the 
same way as if the driver were human. 

Naturally, some moral dilemmas may be involved in these choices 
and decisions: for example, whether it is legitimate for the ar system to 
make decisions in matters of human life, and whether it is legitimate for 
AI systems to cause human death or severe injury. But these dilemmas are 
no different from the moral dilemmas of self-defense, discussed earlier. 1? 
Moreover, moral questions are not to be taken into consideration in rela- 
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tion to the question of criminal liability. Consequently, it appears that the 
general defense of necessity is applicable to AT systems in a similar way 
to self-defense. 


5.3.3. Duress 

Can an AI system be considered as acting under duress in the criminal 
law context? Duress is a justification of the same type as self-defense and 
necessity. All are designed to mitigate the society’s absolute monopoly 
on power, already discussed. The principal difference between self- 
defense, necessity, and duress is in the course of conduct. In self-defense, 
the defender’s reaction is intended to repel the attacker; in necessity, it is 
a reaction against an external innocent object; and in the case of duress, it 
is surrendering to a threat by committing an offense. 

For example, a retired criminal with expertise in breaking into safes is 
no longer active. He is invited by ex-friends to participate in another rob- 
bery, where his expertise is required. He says no. They try to convince him, 
but he still refuses. So they kidnap his daughter and threaten him that if 
he does not participate in the robbery, they will kill her. He knows them 
well, and knows that they are capable of doing it. He also knows that if he 
involves police, they will kill his daughter. As a result, he surrenders to 
the threat, participates in the robbery, and uses his expertise. If captured, 
he can claim duress. Self-defense and necessity are irrelevant in this case 
because he surrendered to the threat rather than face it. 

The traditional approach toward duress is that under the right circum- 
stances, it can justify the commission of offenses.’ The traditional justifi- 
cation of the defense is the understanding in criminal law of the weakness 
of human nature. At times, people would rather commit an offense under 
threat than face the threat and pay the price by causing harm to precious 
interests. Until the eighteenth century, the general defense of duress was 
applicable to all offenses." Later, the Anglo-American legal systems nar- 
rowed its applicability, and today it no longer includes severe homicide 
offenses that require mens rea.M6 

In the safe-breaking example, if the offense were not robbery but 
murder, the general defense of duress would not be applicable for the im- 
position of criminal liability, but only as a consideration in punishment. 
The reason for the narrow applicability is the sanctity of human life." But 
there are many exceptions to this approach." In general, with the narrow 
exception for homicide, duress enjoys broad application as a general de- 
fense in criminal law worldwide. The individual who acts under duress is 
considered to be choosing the lesser of two evils, from his or her point of 
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view: the evil of committing an offense by surrendering to the threat versus 
the evil of harming a legitimate interest. 

Given that the act performed under duress is the individual’s, carried 
out when the authorities are not available or effective, the general de- 
fense of duress partially reduces the applicability of the general concept 
whereby only society (the state) has a monopoly on power, as discussed 
earlier. Being in a situation that requires an act of duress is considered to 
nullify the individual’s fault required for the imposition of criminal li- 
ability. The modern basis of this defense is the understanding that an indi- 
vidual may surrender to a threat rather than face it because not all people 
are heroes, and no one is required by law to be a hero.’ 

There are several conditions for this general defense to be applicable: 


1. The protected interest must be legitimate. Legitimate interests 
are the life, freedom, body, and property of the individual or 
of other individuals. No previous introduction between the 
person threatened and the defender is required.” 

. The protected interest must be in clear and imminent danger." 

. The act of duress must be in surrender to a threat. 

. The act of duress must be proportional to the danger.’ 

. The defender must not be in a situation to control the danger or 
the conditions for its occurrence (actio libera in causa). 
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If all these conditions are met, the individual is considered to be acting 
under duress, and therefore no criminal liability is imposed on him or her 
for the commission of the offense. But not every person surrendering to a 
threat can be considered to act under duress; only when the act meets the 
listed conditions in full. The question is whether the general defense of 
duress is applicable to ar systems. The answer depends on the capabili- 
ties of the a1 systems to meet the conditions just outlined, in the relevant 
situations. 

Four of the listed conditions—numbers 1, 2, 4, and 5—are almost 
identical with those of self-defense and necessity, mutatis mutandis; the 
difference is that in most legal systems, duress does not require an im- 
mediate act because the threat and danger to the legitimate interest may 
be continuous. The main difference between self-defense, necessity, and 
duress lies in one of the conditions. In self-defense, the act is directed 
toward the attacker; in the case of necessity, the act is directed toward an 
external or innocent interest; and acting under duress is surrender to a 
given threat. 

The question concerning A1 systems in this context is whether they are 
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capable of choosing the lesser of two evils. For example, a prison-guard 
robot captures an escaping prisoner, who points a loaded gun at a human 
prison guard and threatens that if he is not released immediately by the 
robot, he will shoot the human guard. The robot calculates the probabili- 
ties and determines that the danger is real. If the robot surrenders to the 
threat, the guard’s life is saved, but an offense is committed and the robot 
becomes an accessory to escape. If the robot does not surrender, no offense 
is committed, the escape is foiled, but the human guard is killed. 

If the guard who captured the prisoner were human, then no criminal 
liability would be imposed on him owing to the general defense of duress 
because all the conditions of this defense are met. An AT system can calcu- 
late the probabilities for each option and choose the possibility with the 
lowest number of casualties. Strong A1 systems are already used to predict 
complicated events (for example, weather patterns), and calculating the 
probabilities in the prison-guard example is considered much easier. The 
AI system analyzing the possibilities is faced with the same two possibili- 
ties that the human guard has to consider. 

If the al system takes into account the number of probable casualties, 
based on its programming or as a result of the relevant machine learn- 
ing, it will probably choose to surrender to the threat. This choice by the 
AI system meets all the conditions of duress, so that if it were human, no 
criminal liability would be imposed on it owing to the general defense of 
duress. Should the al system be treated differently, then? Moreover, if the 
Al system chooses the alternative option, and causes not the lesser but the 
greater of two evils to occur, we would probably want to impose criminal 
liability (on the programmer, the user, or the al system) in exactly the same 
way as if the prison-guard robot were human. 

Naturally, some moral dilemmas may be involved in these choices and 
decisions: for example, whether it is legitimate for the al system to make 
decisions in matters of human life, and whether it is legitimate for AT sys- 
tems to cause human death or severe injury. But these dilemmas are no 
different from the moral dilemmas of self-defense and necessity, discussed 
earlier.* Moreover, moral questions are not to be taken into consideration 
in relation to the question of criminal liability. Consequently, it appears 
that the general defense of duress is applicable to ar systems in a similar 
way to self-defense and necessity. 


5.3.4. Superior orders 
Is an al system on official duty, committing an offense under orders from 
its superior, protected from criminal liability? The general defense of su- 
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perior orders is relevant for individuals who discharge official duties in 
authoritarian hierarchical official organizations such as the army, police, 
and rescue forces. These individuals are often reguired to act against their 
natural instincts. When faced with a gigantic fire, one's natural instinct is 
to escape, not to walk into it and save trapped people. One of the strongest 
factors in the success of such missions is discipline. Disciplined soldiers 
are likely to fulfill their mission even if this involves risking their lives. 
Therefore, the first thing that soldiers are taught is discipline. 

But at times, orders from superiors contradict criminal law, and carry- 
ing them out involves committing an offense. There are two extreme models 
for a solution to this problem. The first is the absolute defense model, in 
which all actions under superior orders are protected.!”° This model places 
discipline above the rule of law, and only the commanders are to be held 
criminally liable according to this model. The other extreme model is the 
absolute responsibility model, in which no action is protected from crimi- 
nal liability, even if performed under superior orders. This model places 
the rule of law above discipline. 

Under the first extreme model, the individual has no discretion in com- 
mitting the offense. This model has led to the commission of war crimes 
and crimes against humanity during World War II and other wars. Under 
the second extreme model, the individual must be an expert in criminal 
law or accompanied by an attorney at all times to make sure of not being 
criminally liable for actions ordered by superiors. The disadvantages of 
these models have led to the emergence of moderate models, the most 
common of which is the manifestly illegal order model. According to this 
model, the individual is protected from criminal liability unless he or she 
performs a manifestly illegal order; if the individual performs an illegal 
order that is not manifestly illegal, he or she is protected.127 

The ultimate question concerns the difference between illegal and 
manifestly illegal orders. Both types objectively contradict the law, but 
the manifestly illegal order contradicts the public policy as well. Every so- 
ciety follows a public policy that reflects its common values. These values 
may be moral, social, cultural, religious, and so on. Different societies em- 
brace different values and have differing public policies. A manifestly il- 
legal order is one that harms these values, and thus harms society’s public 
policy and its self-image. 

For example, a soldier is ordered to rape a civilian who resists the mili- 
tary operation. Rape is illegal, and in this situation it is manifestly illegal 
because it violates the basic values of modern Western society. There is no 
accurate and conclusive examination to distinguish between illegal and 
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manifestly illegal orders because public policy is dynamic, and it changes 
with time, population, and social trends. But public policy may be taught, 
usually inductively, case to case. Consequently, two main conditions must 
be met for the general defense of superior orders to be applicable: 


1. hierarchical subordination to authorized public authority; and 
2. superior orders that reguire obedience and are not manifestly 
illegal. 


If both conditions are met, the individual is considered to be acting under 
superior orders, and therefore no criminal liability is imposed on him or 
her for the commission of the offense. But not every time an individual 
obeys superior orders is the general defense applicable; only when the act 
meets the listed conditions in full. The question is whether the general 
defense of superior orders is applicable to ar systems. The answer to this 
question depends on the capabilities of the a1 systems to meet the condi- 
tions just outlined, in the relevant situations. 

The first condition has to do with the objective characteristics of the 
relationships between the individual and the relevant organization.’ 
This relationship must involve a hierarchical subordination to authorized 
public authority and a legitimate and operative system of hierarchical 
order. Such systems exist in the army, police, and so on. Private organiza- 
tions, however, have no authority to commit offenses. AI systems are in use 
in many of these organizations, including the military, police, and prisons, 
and are operated under superior orders for their regular activity. The sys- 
tems perform various tasks under these orders. 

The second condition has to do with the characteristics of the given 
superior order. The order must require obedience; otherwise, it cannot 
be considered an order. As far as its content is concerned, the order must 
not be manifestly illegal. If the order is legal or illegal, but not manifestly 
illegal, it satisfies this condition. Classification of the order as illegal or 
manifestly illegal is at the discretion of the court, but it may be learned 
inductively based on analysis of individual cases. al systems equipped 
with machine learning facilities are capable of inferring at least the general 
outlines of manifestly illegal orders.1”° 

For example, an ai drone operated by the air force is on a mission to 
seek out a terrorist lab and destroy it. The drone finds its target, and then 
delivers the information to headquarters. The information indicates that 
the lab is populated by a known terrorist and his family. The drone re- 
ceives the order to attack the lab using a heavy bomb. After calculating the 
probabilities, the drone determines that all the people in the lab will die, 
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and then it executes the order. When the records of the drone are exam- 
ined, it emerges that the drone understood the legality of the orders to be 
in a grey area regarding the terrorist’s family, and that it could have flown 
lower and destroyed the lab with fewer casualties. 

If the drone were human, it would probably claim the general defense 
of superior order, because international law accepts such orders under 
certain situations, as long as the order is not manifestly illegal. Therefore, 
a human pilot would probably have been acquitted in such a case, and 
the general defense would have been applicable. Should an ai system be 
treated differently? If both the human pilot and the ai system have the 
same functional discretion, and both meet the conditions of this general 
defense, there is no legitimate reason for using a double standard in such 
cases. 

Naturally, the criminal liability of the a1 system, if any, does not affect 
the criminal liability of its superiors, if any (in case of an illegal order). 
Some moral dilemmas may also be involved in these choices and deci- 
sions: for example, whether it is legitimate for the a1 system to make deci- 
sions in matters of human life, and whether it is legitimate for a1 systems 
to cause human death or severe injury. But these dilemmas are no different 
from the moral dilemmas involving the applicability of other general de- 
fenses, discussed earlier. Moreover, moral questions are not to be taken 
into consideration in relation to the question of criminal liability. Conse- 
quently, it appears that the general defense of superior orders is applicable 
to AI systems. 


5.3.5. De minimis 

Can the general defense of de minimis be applicable for Ar systems? In 
most legal systems, offenses are defined and formulated broadly, inevita- 
bly resulting in over-inclusion or over-criminalization, so that cases that 
are not supposed to be considered criminal end up being included within 
the scope of the offense. At times, the criminal proceedings in these cases 
would cause more social harm than benefit. For example, the case of a 
fourteen-year-old girl who steals her brother’s basketball falls within the 
scope of the offense of theft, and the question is whether it would be ap- 
propriate to institute criminal proceedings for theft in such a case, consid- 
ering its social consequences. 

Most legal systems solve such problems by granting broad discretion to 
the prosecution and the courts. The prosecution may decide not to initiate 
criminal proceedings in cases of low public interest, and if such proceed- 
ings are started, the court may decide to acquit the defendant owing to low 
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public interest. When the prosecution exercises this option, it is within its 
administrative discretion. When the court exercises this discretion, it is 
within its judicial power through the application of the general defense of 
de minimis, which enables the court to acquit the defendant based on low 
public interest in the case. 

This type of judicial discretion has been widely accepted since ancient 
times. Roman law, for example, determined that criminal law does not 
extend to infant and petty matters (de minimis non curat lex), and that 
the judge should not be troubled with such matters (de minimis non curat 
praetor). In modern criminal law, the general defense of de minimis is 
seldom exercised by the court given the wide administrative discretion 
of the prosecution. But in some extreme cases, the court may exercise 
this judicial discretion, in addition to the administrative discretion of the 
prosecution.'*? 

The basic test for de minimis defense is one of the social endangerment 
involved in the commission of the offense. For the general defense of de 
minimis to be applicable, the offense should reflect extremely low social 
endangerment. Naturally, different societies at various times may attri- 
bute different levels of social endangerment to the same offenses, because 
social endangerment is dynamically conceptualized through morality, cul- 
ture, religion, and so on. The relevant social endangerment is determined 
by the court. The question is whether this general defense is relevant for 
offenses committed by al systems. 

Applicability of de minimis defense depends on the particulars of the 
case at hand and is based on its relevant aspects, and not necessarily on 
the identity of the offender. The personality of the offender may also be 
taken into consideration, but only as part of assessing the case. For this 
reason, there is no difference between humans, corporations, or AT entities 
regarding the applicability of this defense. The required low social endan- 
germent is reflected in the factual event (in rem). For example, a human 
driver swerves on the road and hits the guardrail. No damages are caused 
to the guardrail or to other property, and there are no casualties. This is 
a case for the de minimis defense, although it may be within the scope of 
several traffic violations. 

Would this case be legally different if the driver were not human but 
an AI entity? And would it be different if the car were owned by a corpo- 
ration? There is no substantive difference between humans, corporations, 
and A1 systems regarding the applicability of the de minimis defense, es- 
pecially in light of the fact that this general defense is based on the char- 
acteristics of the factual event and not necessarily on those of the offender. 
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Consequently, it appears that the general defense of de minimis is appli- 
cable to al systems. 


closing The opening exaMple: r oboT Killin g in self- defense 


We opened this chapter with the example that described an AT robot, func- 
tioning as a prison guard, who killed in self-defense. Returning to this 
example, we can analyze it based on the insights gained from this chapter. 
Let us assume that after examining the robot’s records, it transpires that 
the robot had consolidated awareness of the standards of criminal law 
regarding the factual components of the homicide. There is no doubt that 
the human prison guard died as a result of being smashed against the wall 
by the robot (factual causal connection). Following through this example 
step by step, we may find several applicable general defenses. 

First, the virus planted in the main computer interfered with the ro- 
bot’s ability to create the right image of factual reality, and caused it to 
ignore some of the rules of the system. Two types of general defenses may 
be applicable as a result. The first is factual mistake, discussed previ- 
ously. The reason for the factual mistake is immaterial for its legal con- 
sequences, and therefore, if during the event the factual image of the 
Al system differed from what actually happened because of the effect of 
the virus, this can serve as the basis for the applicability of factual mistake 
general defense. 

The effect of the virus on the cognitive or volitive malfunction of the 
robot during its operation should also be examined based on the records 
of the ar system. If the virus caused the malfunction, then this can be the 
basis for the applicability of another general defense having to do with loss 
of self-control, insanity, or intoxication. The choice between these options 
depends on the concrete effect of the virus on the ai robot. Because the 
planting of the virus was involuntary (done by hackers), the consequences 
of the effects of the virus will most likely prevent imposition of criminal 
liability if the conditions of the relevant defense are met. 

When the robot apprehended the two struggling humans, it misinter- 
preted the situation as one in which two prisoners were struggling. This 
misinterpretation can be proven by analyzing the robot’s records. If the 
robot indeed misinterpreted the situation, whether or not as a result of 
the virus, the general defense of factual mistake is applicable. The same 
is true for the robot misinterpreting the command to keep away as if this 
had been a threat to its mission. If the robot indeed interpreted this com- 
mand as a threat, whether or not it was a threat in reality, can the reaction 
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of grabbing the human guard and smashing him against the wall be justi- 
fied and defended? 

As understood by the robot, this reaction is at most one of repelling a 
threat. If all the conditions of self-defense are met, this may serve as the 
basis for the general defense of self-defense. The robot’s mission, however, 
was to prevent the escape of other prisoners but not to protect their lives, 
freedom, body, or property. Therefore, self-defense is inapplicable in this 
case, as are necessity and duress. Even if the system had orders to kill re- 
sisting prisoners, the general defense of superior orders would not have 
been applicable. Killing prisoners in response to an attempted escape, that 
results in the escape of other prisoners (the disguised prisoner managed 
to escape as a result of the robot’s actions), would likely be considered as 
following a manifestly illegal order, obeying which is not protected. 

But if it can be proven, based on the records, that the robot’s decision 
to kill the guard was a direct consequence of the effect of the virus on the 
system, then the legal situation may be different. If the virus caused the 
robot to act automatically when killing the guard, or caused a malfunction 
in its cognitive or volitive processes as a result of which the robot killed 
the guard, then the general defenses of loss of self-control, insanity, or in- 
toxication may be applicable according to the concrete effect of the virus. 

Naturally, this does not reduce the criminal liability of others involved 
in the event, including humans (superiors, prisoners, guards, and hackers), 
corporations (the prison service and the manufacturer of the ar systems), 
and other ar systems. It appears, however, that general defenses in criminal 
law are applicable to A1 systems as well as to humans and corporations. 


Sen Tencing ai 


Is it possible to impose a prison sentence on an AT robot? How can such a 
punishment be carried out in practice? One of the objectives of the crimi- 
nal process is sentencing. The legal discussion taking place in court re- 
garding the criminal liability of the defendant is often considered to be a 
preliminary discussion before sentencing. Debating the criminal liability 
of the defendant is not a theoretical exercise, but it has practical conse- 
quences. If the defendant is acquitted, no punishment is imposed. But if 
he or she is convicted, the punishment must reflect the attitude of society 
toward the commission of the offense and toward the personal circum- 
stances of the defendant. 

The question is whether the sentencing process in criminal law and 
criminal punishments are relevant to machines. For example, after all 
the factual and mental element requirements of manslaughter were met 
by an AT robot, the court convicted the robot of manslaughter, and the 
prosecution and defense presented evidence for punishment. Under the 
circumstances, if the robot had been a human, the court would have sen- 
tenced this offender to three years of imprisonment. Is such a punishment 
applicable to a robot? Is criminal punishment applicable to machines 
at all? 

This question is relevant and complementary to the examples that 
began each of the previous chapters. To answer it, we must first explore 
the general purposes of sentencing and their applicability to AT entities, 
and then understand the legal technique of imposing punishments on non- 
human entities. Finally, we will discuss the applicability of punishing 
Al entities. 
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6.1. concenTual applicabili Ty of cri Minal 
punish Ment of ai entities 


6.1.1. ai purpose of Sentencing: 
combining r ehabilitation and incapacitation 

Criminal punishments are imposed for a purpose. There are four common 
general purposes of sentencing, which also function as the general sentencing 
considerations: retribution, deterrence, rehabilitation, and incapacitation. 

Retribution (lex talionis) is the most ancient purpose of sentencing.’ It 
has to do with the past, and contains many aspects of revenge. The basic 
idea of retribution is expressed in the maxim “an eye for an eye,” that 
is, whatever you have caused shall be imposed on you. Retribution pur- 
posefully causes suffering to the offender to a degree that fits the suffering 
caused by the offender, and not more. Retribution functioned as a lenient 
form of punishment because it prevented society from causing greater 
suffering to offenders than they have caused to others.* For example, the 
death penalty would not be appropriate for breaking someone’s tooth. 

The importance of retribution as a major objective of punishment grad- 
ually diminished between the 1920s and the 1970s, primarily because 
of criticism that retribution was archaic and retrospective, and therefore 
did not sufficiently take into account the social benefit of punishment.* 
The purpose of retribution was considered barbaric. Nevertheless, in light 
of the actual and conceptual failure of rehabilitation, as we will discuss, 
since the 1970s retribution has regained a significant position among the 
purposes of punishment, in its modern form of “just desert.”® 

Whereas rehabilitation concentrates on the personal characteristics 
of the offender, thereby punishing the offender for his or her character, 
retribution concentrates on the characteristics of the offense.’ The “just 
desert” approach incorporates some personal aspects of the offender into 
the retribution, thereby softening it.” Thus, there are two important factors 
in evaluating the offender’s punishment under retribution: (1) the factual 
damage caused by the offender to society, and (2) the personal culpability 
of the offender.® Personal culpability is measured mostly by the mental 
element of the offender." 

The question for our discussion is whether retribution is relevant to 
Al systems. Retribution is intended primarily to satisfy society. Causing 
suffering to the offender, in itself, has no prospective value. The suffering 
may deter the offender, but that is part of the general purpose of deter- 
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rence, not of retribution. Retribution may have a cathartic effect on soci- 
ety and the victims by causing the offender to suffer, but in this context, 
punishing machines by retribution would be meaningless and impractical. 

Some people, when they are in a great hurry and their car will not 
start, may become angry and hit the car, kick it, or yell at it. Punishing 
machines, including highly sophisticated a1 robots, by retribution would 
be the same as kicking a car. Some people may achieve a certain release of 
their anger, but nothing more would be accomplished. Machines do not 
suffer, and as long as retribution is based on suffering, it is not relevant to 
punishing robots. This is true for both classic and modern (“just deserts”) 
approaches to retribution. 

Further, if retribution functions as a lenient form of sentencing aimed 
at preventing revenge, its irrelevance to AI sentencing is even more pro- 
nounced. Revenge is assumed to cause more suffering to the offender than 
would the official punishment, but because machines do not experience 
suffering, the distinction between revenge and retribution is meaningless 
for them. 

Deterrence is the teleological response to deontological retribution. 
Deterrence emerged as the general purpose of sentencing toward the end of 
the eighteenth century. Inspired by utilitarianism, legal scholars searched 
for the prospective value of the law. This search included sentencing as 
well, and it was assumed that criminal punishment may discourage of- 
fenders and potential offenders from committing further offenses. Some 
scholars called for the inclusion of punishment components as integral 
parts of the definition of offenses." Deterrence reflected the human wish 
for a better society, without delinquency.” 

This better society was supposed to be achieved by intimidation. Of- 
fenders and potential offenders, afraid of punishment, were expected to re- 
consider their delinquent thoughts and eventually renounce their criminal 
plan. Because punishment was supposed to deter people from committing 
offenses, it was assumed that the more severe the punishment, the more 
people would avoid committing offenses. But this assumption turned out 
to be misleading for two reasons. 

First, people are deterred more by efficient enforcement than by pun- 
ishment.’* For example, the effect on driving habits of doubling the maxi- 
mum penalty from $100 to $200 for exceeding the speed limit on a road 
that has practically no police presence would be insignificant. Yet if more 
police officers patrolled the same road, even if the fine were lowered to 
$80, drivers would be more deterred from speeding. When city inspectors 
are on strike and no citations are issued for parking in restricted areas, 
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most people tend to park their cars in these areas, regardless of the level 
of maximum punishment. Thus, the deterrent value of punishment alone 
is very limited. 

Second, the deterrent value of punishment, even if it is maximal, tends 
to erode in time with use, and at some point, the punishment may end 
up functioning as an incentive for repeated offense.1* For instance, when 
an offender is sentenced to prison for the first time, the deterrent effect 
of the punishment is at its maximum level for that offender. The second 
time, the effect is reduced, and the tenth time it is reduced dramatically. 
After spending forty years in prison, the prisoner does not really have a 
life outside of prison. Prison provides meals, social status, social security, 
and friends. If released, the prisoner may commit offenses only to return 
to this familiar social environment. 

Thousands of homeless people in the Western world are committing 
offenses on cold days to find shelter from the cold, have a warm meal, 
and find a place to sleep. Not only does punishment not serve as a deter- 
rent, but it serves as an incentive to repeat offending. In other words, the 
punishment turns into the benefit of the offense. Nevertheless, for most 
offenders and potential offenders, a combination of proper enforcement 
with appropriate punishments can supply some deterrence, even if this is 
not entirely effective. Our question, then, is whether deterrence is relevant 
to AI systems. 

Deterrence is intended to prevent the commission of the next offense, 
by using intimidation. In the current technology, intimidation is a feeling 
that machines cannot experience. Intimidation is based on a fear of future 
suffering imposed if an offense is committed. Because machines do not 
experience suffering, as we have already noted, both intimidation and the 
reason for it are nullified when considering appropriate punishment for 
robots. At the same time, both retribution and deterrence may be relevant 
purposes for the punishment of human participants in the commission of 
offenses by ai entities (for example, users and programmers). 

Rehabilitation is a relatively newer purpose of modern sentencing, 
based on the idea that the punishment and the process of sentencing may 
serve as opportunities for correcting offenders’ social problems. Rehabili- 
tation was inspired by the religious concept that every person may be 
corrected given the right conditions, and has been embraced as a major 
sentencing consideration since 1895 in Britain, spreading to the rest of 
the world.’ The golden age of rehabilitation was between the 1920s and 
1970s,” and its popularity rested on its prospective quality that offered 
hope for a better world. 
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Enthusiasm for rehabilitation waned with the publication of the Mar- 
tinson report in 1974.'* The publication examined rehabilitation reports 
in the state of New York and came to the surprising conclusion that “noth- 
ing works.” The conclusions were reexamined repeatedly, and each time 
confirmed,” giving rise to various criticisms of rehabilitation, on the part 
of research and even popular art.” Disappointment with rehabilitation 
triggered a revival of retribution, in its modern form as “just desert,” as 
already noted. Rehabilitation was not abolished entirely, however, and 
it is still being considered in the sentencing process, but more carefully. 

The prospective purpose of rehabilitation is to address the roots of 
delinquency. By solving the problems that lead to it, delinquency may be 
solved, or at least dramatically reduced. To this end, the court must un- 
derstand the causes of an offender’s delinquency, assess the potential for 
rehabilitation (using such professionals as social workers and psycholo- 
gists), and eventually decide on a proper punishment aimed at solving the 
delinquency problem.” One of the most popular trends in rehabilitation 
is to grant the offender the cognitive and social skills needed to cope with 
daily life.*° 

These tools are supposed to cause the offender to make inner changes 
in relation to his or her social, family, or professional environment, provid- 
ing an opportunity for resocialization and reintegration into society and 
for leaving delinquency behind, to benefit the offender and also society.” 
Our question is whether rehabilitation is relevant to al systems. 

AI systems can experience decision-making processes and reach de- 
cisions that may appear unreasonable, as we have discussed.” At times, 
an AI system may need external guidance in order to refine its decision- 
making process, which may be part of the machine-learning process. For 
Al entities, rehabilitation functions in exactly the same way as for humans, 
causing them to make better decisions in their everyday lives from the 
point of view of society. Following this approach, the punishment of Ar 
entities would be directed at refining the machine-learning process. 

Having been rehabilitated, an al system would be able to form better 
and more accurate decisions, by applying more limitations to its discretion 
and by refining the process through machine learning. Thus, the punish- 
ment, if correctly applied to an individual AT system, would become part of 
the machine-learning process. Directed by the rehabilitative punishment, 
the al system would therefore have better tools to analyze factual data. 
This is the same effect that rehabilitative punishment is intended to have 
on humans by granting them better tools for facing factual reality. 

Consequently, rehabilitation can be a relevant purpose of punishment 
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for al systems because it is not based on intimidation or suffering, but is 
directed at improving the performance of the AT system. For humans, this 
consideration may be secondary in many cases, but for al systems it may 
be a primary purpose of punishment. Nevertheless, rehabilitation is only 
one of the considerations that are relevant to al systems; incapacitation is 
another. 

Incapacitation is the physical prevention of offenders from committing 
further offenses. This general consideration is based on skepticism toward 
the effectiveness of the other prospective considerations (deterrence and 
rehabilitation) to reduce delinquency rates; it focuses therefore on the 
physical capabilities of delinquency. The basic approach is to neutralize 
the offender’s physical capabilities to commit offenses.”° For example, in- 
capacitation is the general consideration that justifies the chemical castra- 
tion of sex offenders by eliminating their sexual impulse and their ability 
to commit further sexual offenses. 

For the physical prevention of delinquency, it is not necessary to 
bring about inner changes or cognitive corrections in the offender’s mind. 
The offender is not required to change his or her personality or values 
for this consideration to be effective.” In some ways, the consideration 
of incapacitation is the expression of disappointment, even despair with 
human nature, contrary to rehabilitation, which expresses hope. When 
society feels hopeless toward an offender’s willingness or mental capabil- 
ity to stop the delinquent activity, both deterrence and rehabilitation may 
appear to be ineffective; but further offense must still be prevented, so 
society resorts to incapacitation. 

In this way, regardless of whether the offender understands or agrees 
with the demands of society, the next offense is prevented because the 
physical capability to commit the offense is neutralized. If in most cases, 
rehabilitation is considered a lenient general consideration of punish- 
ment, incapacitation is considered a harsh one. Our question is whether 
incapacitation is relevant to AI systems. 

If an al system commits offenses and lacks the capability to change its 
ways through machine learning, only incapacitation can supply an ad- 
equate answer. Regardless of whether the A1 system understands the mean- 
ing of its activity, or whether the ai system is equipped with proper tools to 
perform inner changes, delinquency must still be prevented. In a situation 
of this type, society must disable the physical capabilities of the A1 system 
to commit further offenses, despite its other skills. This is what the society 
does with human offenders in similar cases.” 

We can conclude that two relevant considerations for the punishment 
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of AT systems are rehabilitation and incapacitation. Both reflect extreme 
poles of sentencing, and both serve the purposes of criminal law regard- 
ing nonhuman offenders. When the AI system is capable of carrying out 
inner changes that affect its activity, rehabilitation appears to be preferable 
to incapacitation, but when it does not possess such capabilities, inca- 
pacitation is preferable. Thus, the punishment is adjusted to the relevant 
personal characteristics of the offender, as in the case of human offenders. 


6.1.2. The | egal Technique of c onversion: 

The case of corporations (r ound 2) 
We have seen that sentencing considerations are relevant to AI systems. 
The question is, how can we impose human punishments on machines? 
For example, how can we impose imprisonment, fines, or capital punish- 
ment on AI systems? To do so, we need a legal technique of converting 
human penalties into penalties for AT entities. Such technique may be in- 
spired by the one used to convert human penalties into penalties suitable 
for corporations. The first time we examined corporations (Round 1), we 
were concerned with the idea of imposing criminal liability on AI sys- 
tems.” This time (Round 2) we are concerned with the idea of imposing 
punishments on these systems. 

Not only has criminal liability been imposed on corporations for cen- 
turies, but corporations have also been sentenced, and not only to fines. 
Corporations are punished in various ways, including imprisonment. Note 
that corporations are punished separately from their human officers (di- 
rectors, managers, employees, and so on) in exactly the way that criminal 
liability is imposed on them separately from the criminal liability, if any, 
of their human officers. There is no debate over the question whether cor- 
porations should be punished using a variety of punishments, including 
imprisonment—the question concerns only how to do it.” 

To answer this question of “how,” a general legal technique of conver- 
sion is needed, as just noted. This operation is carried out in three prin- 
cipal stages. First, the general punishment itself (for example, imprison- 
ment, fine, probation, or death) is analyzed regarding its roots of meaning. 
Second, these roots are sought in the corporation. Third, the punishment 
is adjusted according to the roots found in the corporation. For example, 
in the case of imposition of incarceration on corporations, first incarcera- 
tion is traced back to its roots in the act of depriving individuals of their 
freedom, and then a meaning is sought for the concept of freedom for 
corporations. 

After this meaning has been understood, in the third and final stage, 


sentencing ai - 163 


the court imposes a punishment that is the equivalent of depriving a cor- 
poration of its freedom. This is how the general legal technique of conver- 
sion works in the sentencing of corporations. At times, this requires the 
court to be creative in the adjustments needed to make punishments ap- 
plicable to corporations, but the general framework is clear and workable, 
and it has been implemented with all types of punishments imposed on 
all types of corporations.” 

An excellent example is the case of the Allegheny Bottling Company,” 
a corporation found guilty of price-fixing (antitrust). It was agreed that 
under the given circumstances, if the defendant were human, the appro- 
priate punishment would be imprisonment for a certain term. The ques- 
tion concerned the applicability of imprisonment to corporations—in 
other words, a question of “how.” As a general principle, the court de- 
clared that it “does not expect a corporation to have consciousness, but it 
does expect it to be ethical and abide by the law.”** 

The court did not find any substantive difference between humans 
and corporations in this matter, adding that “[t]his court will deal with 
this company no less severely than it will deal with any individual who 
similarly disregards the law.”** This statement reflects the basic principle 
of equalizing punishments of human and corporate defendants. In this 
case, the corporation was sentenced to three years’ imprisonment, a fine of 
$1 million, and probation for a period of three years. The court proceeded 
to discuss the idea of corporate imprisonment based on the three stages 
just described. 

First, the court asked what were the general meanings of imprison- 
ment, and accepted the definitions of imprisonment as “constraint of a 
person either by force or by such other coercion as restrains him within 
limits against his will” and as “forcible restraint of a person against his 
will.” The court’s conclusion was simple and clear: “[t]he key to corpo- 
rate imprisonment is this: imprisonment simply means restraint” and 
“restraint, that is, a deprivation of liberty.” The court’s conclusion was 
reinforced by several provisions of the law and also of case laws. Conse- 
quently, “[t]here is imprisonment when a person is under house arrest, for 
example, where a person has an electronic device which sends an alarm if 
the person leaves his own house.” 

This ended the first stage. In the second stage, the court searched for 
a meaning of this punishment for corporations, concluded that “[c]orpo- 
rate imprisonment requires only that the Court restrain or immobilize the 
corporation,” and proceeded to implement the prison sentence on the 
corporation according to this insight. Thus, in the third and final stage, 
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the court made imprisonment applicable to the corporations and imple- 
mented it as follows: 


Such restraint of individuals is accomplished by, for example, 
placing them in the custody of the United States Marshal. Like- 
wise, corporate imprisonment can be accomplished by simply 
placing the corporation in the custody of the United States Mar- 
shal. The United States Marshal would restrain the corporation 
by seizing the corporation’s physical assets or part of the assets 
or restricting its actions or liberty in a particular manner. When 
this sentence was contemplated, the United States Marshal for the 
Eastern District of Virginia, Roger Ray, was contacted. When asked 
if he could imprison Allegheny Pepsi, he stated that he could. 
He stated that he restrained corporations regularly for bankruptcy 
court. He stated that he could close the physical plant itself and 
guard it. He further stated that he could allow employees to come 
and go and limit certain actions or sales if that is what the Court 
imposes. 

Richard Lovelace said some three hundred years ago, “stone 
walls do not a prison make, nor iron bars a cage.” It is certainly true 
that we erect our own walls or barriers that restrain ourselves. Any 
person may be imprisoned if capable of being restrained in some 
fashion or in some way, regardless of who imposes it. Who am I 
to say that imprisonment is impossible when the keeper indicates 
that it can physically be done? Obviously, one can restrain a corpo- 
ration. If so, why should it be more privileged than an individual 
citizen? There is no reason, and accordingly, a corporation should 
not be more privileged. 

Cases in the past have assumed that corporations cannot be 
imprisoned, without any cited authority for that proposition. .. . 
This Court, however, has been unable to find any case which ac- 
tually held that corporate imprisonment is illegal, unconstitu- 
tional or impossible. Considerable confusion regarding the ability 
of courts to order a corporation imprisoned has been caused by 
courts mistakenly thinking that imprisonment necessarily involves 
incarceration in jail. . . . But since imprisonment of a corporation 
does not necessarily involve incarceration, there is no reason to 
continue the assumption, which has lingered in the legal system 
unexamined and without support, that a corporation cannot be 
imprisoned. Since the Marshal can restrain the corporation’s lib- 
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erty and has done so in bankruptcy cases, there is no reason that 
he cannot do so in this case as he himself has so stated prior to the 
imposition of this sentence.*” 


Thus, imprisonment can be applied not only to human but also to 
Corporate offenders. Following the same approach, imprisonment is not 
the only penalty applicable to corporations, but other penalties can be 
converted as well, even if they were originally designed for human offend- 
ers. And if this is true for imprisonment, which is an essentially human 
penalty, fine can be easily collected from corporations, just like taxes. This 
raises the equivalent question regarding AI systems. 

Using the general legal technique of conversion, as just presented and 
applied to corporate delinquency, human punishments can be made ap- 
plicable in the same way and imposed on al systems as well. Next we ex- 
amine the applicability of common punishments in modern criminal law 
to AI systems. 


6.2. applicabili Ty of punishMenTs To ai enTiTies 


We will review the applicability and imposition of the following punish- 
ments on AI systems: capital punishment, imprisonment, probation, public 
service, and fine. 


6.2.1. capital p unishment 

The death penalty is one of the most ancient punishments in human his- 
tory, and it is considered the most severe penalty in most cultures.** In the 
past it was a common penalty, but since the eighteenth century, the global 
approach has been to restrict and minimize its use. Therefore, capital pun- 
ishment has been replaced by more lenient penalties for many offenses, 
and methods of execution were developed to cause minimal suffering to 
the offenders.*? For example, in the eighteenth century, the length of the 
noose was increased to cause a faster and less painful death, and the guil- 
lotine was introduced for the same reason. 

Cruel methods of execution were prohibited. For example, dismember- 
ing by tying the offender’s arms and legs to running horses was prohibited 
because of the suffering it caused. Some countries abolished capital pun- 
ishment altogether, but most countries did not. The United States Supreme 
Court ruled in 1979 that capital punishment, in the appropriate cases, does 
not infringe on the Eighth Amendment to the Constitution (that is, it is not 
cruel and unusual punishment), and therefore it is constitutionally valid.” 
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The methods of execution used in the United States were not considered 
unconstitutional either.*! 

Retribution justifies capital penalty only for severe offenses in which 
penalty parallels the offense in suffering or result (for example, homicide 
offenses).*? Deterrence can justify capital punishment only as a deterrent 
of the public and not of the offender, because a dead person cannot be 
deterred.** Rehabilitation is completely irrelevant for this punishment, as 
dead people cannot be rehabilitated. And given that retribution and de- 
terrence are irrelevant for al sentencing, as noted previously, and because 
rehabilitation is irrelevant to capital punishment, the only general consid- 
eration that supports capital punishment in the sentencing of A1 entities 
is incapacitation. 

There is no doubt that a dead person is incapacitated from commit- 
ting further offenses and that the most dominant consideration for the 
death penalty is incapacitation.** The question regarding the applicability 
of capital punishment to al systems is how to impose the death penalty 
on them. To answer this question, we must follow the three-stage model 
presented earlier. 

First, we should analyze the meaning of capital punishment; second, 
seek out its roots as far as AI systems are concerned; and finally, adjust the 
punishment to these roots. Functionally, capital punishment is depriva- 
tion of life. Although this deprivation may affect not only the offender but 
other people as well (relatives, employees, and so on), the essence of the 
death penalty is the death of the offender. When the offender is human, life 
means the person’s existence as a functioning creature. When the offender 
is a corporation or an AI system, its life may be defined through its activity. 

A living ar system is a functioning one, therefore the “life” of an AI 
system is its capability to function as such. Stopping the ai system’s ac- 
tivity does not necessarily mean the “death” of the system. Death means 
permanent incapacitation of the system’s life. Therefore, capital punish- 
ment for an AI system means its permanent shutdown so that no further 
offenses or any other activity can be expected on the part of the system. 
When an AT system is shut down by court order, this means that the society 
prohibits the operation of that entity because it is considered too danger- 
ous for society. 

This application of capital punishment on ar systems serves both the 
purposes of capital punishment and of incapacitation (as a general pur- 
pose of sentencing) in relation to AI systems. When the offender is too 
dangerous for society, and society decides to impose the death penalty, if 
the punishment is acceptable in the given legal system, it is intended to 
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accomplish the total and final incapacitation of the offender. This is true 
of human offenders, corporations, and a1 systems. For al systems, perma- 
nent incapacitation means absolute shutdown under court order, with no 
option of reactivating the system again. 

A permanently incapacitated system will not be involved in further 
delinquent events. It may be argued that such shutdown may affect other 
innocent persons (for example, the manufacturer of the system, or its pro- 
grammers and users). Yet this is true not only for ar systems, but for human 
offenders and corporations as well. The execution of an offender also af- 
fects his innocent family (in the case of a human offender) or affects em- 
ployees, directors, managers, shareholders, and so on (in the case of a cor- 
poration). When the offender is an Ar system, shutdown also affects other 
innocent persons, but this is not unique to al systems. Therefore, capital 
punishment may be applicable to ai systems. 


6.2.2. imprisonment and Suspended imprisonment 
Imprisonment is a general term for various penalties. The common charac- 
teristic of these penalties is deprivation of the offender’s liberty. Physical 
incarceration is only one type of imprisonment, although it is the most 
common. For example, under certain circumstances, public service can 
function as imprisonment. The purposes of imprisonment have differed 
in different societies and times. At times, the purpose was to make the of- 
fender suffer; at other times, it was to rehabilitate the offender under tight 
discipline.* But since the eighteenth century, a dominant consideration 
for the evaluation of imprisonment has been its social efficiency.“ 

The social efficiency of the imprisonment is evaluated by the rate of 
recidivism. If the rate of recidivism remains constant or increases, then im- 
prisonment is not considered socially efficient. Therefore, the imposition 
of imprisonment has been used to initiate an inner change in the offender 
for the benefit of society.” Thus, prisoners have been taught professions; 
have learned reading and writing; have been helped to break drugs, alco- 
hol, and violent habits; and have been encouraged to experience working 
during their stay in prison. Imprisonment has been adapted to fit various 
populations, resulting in the creation of such solutions as supermax pris- 
ons* and shock incarceration.*® 

Nevertheless, because imprisonment has become a popular penalty, 
prison overcrowding is currently considered its most acute problem. Retri- 
bution supports imprisonment, especially incarceration, because it makes 
the offender suffer. Deterrence supports imprisonment because the suffer- 
ing caused in prison may deter offenders from recidivism and potential 
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offenders from offending. Both retribution and deterrence, however, are ir- 
relevant to al systems because they experience neither suffering nor fear.*° 
Imprisonment for a1 systems should be evaluated based on rehabilitation 
and incapacitation, which are relevant to AT entities. 

When the offender is deprived of his or her liberty, society can use this 
prison term for the offender’s rehabilitation, by initiating inner change. 
This change can be the result of activity carried out in prison. If the of- 
fender accepts the inner change and does not return to delinquency, then 
imprisonment is considered successful and the offender rehabilitated. At 
the same time, when the offender is under strict supervision inside the 
prison, his or her capability for committing further offenses is dramatically 
reduced, which may be considered incapacitation as long as supervision 
prevents recidivism. 

Here is the question regarding the applicability of imprisonment: how 
can imprisonment be imposed on AI systems? To answer this question, 
we again follow the three-stage model. First, we analyze the roots of the 
meaning of imprisonment; second, we seek the meaning of these roots for 
Al systems; and finally, we adjust the punishment to the roots we have re- 
vealed for al systems. 

Functionally, imprisonment is deprivation of liberty. Although this 
deprivation may affect not only the imprisoned offender, but other people 
as well (for example, relatives, employees, and so on), the essence of the 
imprisonment consists of restricting the offender’s activity. When the of- 
fender is human, liberty means the person’s freedom to act in any desired 
way. When the offender is a corporation or an AT system, its liberty may 
also be defined through its activity. The freedom of an A1 system lies in the 
exercise of its capabilities without restriction, including both the exercise 
of these capabilities and their content. 

Thus, the imposition of imprisonment on AI systems consists of de- 
priving them of their liberty to act by restricting their activities for a spec- 
ified period of time and under strict supervision. During this time the 
AI system may be repaired to prevent the commission of further offenses. 
Repair of the al system may be more efficient if the system is incapacitated, 
especially if it is done under court order. This situation can serve both 
purposes of rehabilitation and incapacitation, which are the relevant sen- 
tencing purposes for A1 systems. When the a1 system is in custody, under 
restriction and supervision, its capability to offend is incapacitated. 

An Al system being repaired through inner changes, initiated by ex- 
ternal factors (for example, programmers working under court order), and 
gaining experience during the period of restriction, can be substantively 
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considered to be in rehabilitation because the chances of the system being 
involved in further delinguency are being reduced. The social value of 
imposing imprisonment on AI systems is real, as the dangerous system is 
being removed from society for repairs, and at the same time, it is inca- 
pable of causing further harm to society. When the process is complete, the 
system may be returned to full activity. If the system is beyond rehabilita- 
tion, then incapacitation becomes the principal objective, dictating a long 
period of imprisonment or even capital punishment. 

A sentence of suspended imprisonment is a conditional penalty." The 
offender is warned that if further offense is committed, the full penalty 
of imprisonment is imposed for the new offense, and in addition, the of- 
fender will have to serve another term of imprisonment for the original of- 
fense. This penalty is intended to deter the offender from offending again, 
for at least as long as the condition remains in force. This penalty is im- 
posed by adding the relevant line to the offender’s criminal record. The 
question of suspended sentences does not concern the execution of the 
penalty, but concerns its social meaning when imposed on a1 systems. 

A sentence of suspended imprisonment for an A1 system is an alert to 
reconsider its course of conduct. This process may be initiated by pro- 
grammers, users, or the manufacturer, in the same way that human of- 
fenders may be assisted by their relatives or professionals (for example, 
psychologists or social workers), and corporate offenders by their officers 
or professionals. A suspended sentence is a relatively lenient measure 
calling for reconsideration of one’s course of conduct. In essence, it is not 
substantially different from imprisonment, although for human offenders 
it may be vastly different because it spares the human offender the suf- 
fering of incarceration and deters him or her from delinquency through 
intimidation. 


6.2.3. probation 

Prison overcrowding has forced states to develop substitutes for incar- 
ceration. One of the popular substitutes is probation, developed in the 
mid-nineteenth century by private charity and religious organizations that 
undertook to care for convicted offenders by giving them the social tools 
they needed to abandon delinquency and reintegrate into society.”2 Most 
countries embraced probation as part of their public sentencing systems. 
The first US state to do that was Massachusetts, in 1878. Thus, probation 
was operated and supervised by the state.” 

The social tools provided to the offender vary from case to case. When 
the offender is addicted to drugs, the social tools include a drug rehabili- 
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tation program. When the offender is unemployed or has no profession, 
the social tools include vocational training. During the time the offender 
is under probation, the authorities supervise and monitor him or her to 
ensure that no further offenses are committed. Probation is dominantly 
rehabilitative, and is appropriate for offenders who have a high potential 
for rehabilitation. Consequently, the court needs an accurate assessment 
of that potential, prepared by the probation service, before sentencing the 
offender to probation." 

Retribution is irrelevant for probation, because probation is not in- 
tended to make the offender suffer. Neither is deterrence relevant to pro- 
bation, which is perceived as a lenient penalty with a negligible deterrent 
value. Moreover, as we have already seen, both retribution and deterrence 
are irrelevant to al sentencing. Incapacitation, which is relevant to AT sen- 
tencing, is not reflected in probation, unless the framework for the specific 
probation is extremely strict and the offender’s delinquent capabilities are 
incapacitated. The dominant purpose of probation, however, is rehabilita- 
tion through the provision of the relevant social skills needed to reinte- 
grate the offender into society. 

Here is the question regarding the applicability of probation: how can 
probation be imposed on al systems? To answer this question, we again 
follow the three-stage model. First, we analyze the roots of the meaning 
of probation; second, we seek the meaning of these roots for Ar systems; 
and finally, we adjust the punishment to the roots we have revealed for 
Al systems. 

Functionally, probation consists of supervising the offender and pro- 
viding him or her with the means to reintegrate into society. These mea- 
sures should match the type of delinquency that was the immediate cause 
for the sentencing.5* Probation works as a functional correction of the of- 
fender. When an offense is committed by an al system, the system must be 
diagnosed to determine whether it can be corrected. At this stage, human 
offenders are diagnosed to determine their potential for rehabilitation. 
Both types of diagnosis are performed by professionals. 

Human offenders may be diagnosed by probation service staff, social 
workers, psychologists, psychiatrists, physicians, and so on. al system of- 
fenders may be diagnosed by technology experts. If the diagnosis shows 
no potential for rehabilitation, then the ultimate purpose of sentencing 
becomes incapacitation, because society wishes to prevent further harm. 
But if the offender has a high potential for rehabilitation, then proba- 
tion and rehabilitation as the purpose of sentencing are taken into con- 
sideration. This is equally true for human offenders, corporations, and 
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Al systems. The core question for Ar system diagnosis is whether the 
system may be repaired in its current configuration, for example, through 
machine learning. 

If the conclusion is that the appropriate penalty is probation, it must 
be applied with reference to the problems that the delinquency raised. 
This is true for all types of offenders. The objective of the treatment is to 
fix a certain problem, therefore the treatment should match the problem. 
The difference between probation and imprisonment, which also includes 
treatment, is the incapacitation of the offender’s delinquent capabilities 
during the period of the penalty. If the offender is not considered danger- 
ous during the period of the treatment, probation may be suitable, but if 
the offender must be incapacitated during treatment, imprisonment may 
be better suited. 

Therefore, if society finds that the Ar system can continue function- 
ing while under treatment, probation may be suitable. If it is too dan- 
gerous for society, imprisonment may be suitable. When probation is fi- 
nally imposed, the ar system must begin the treatment, which consists of 
repair of its inner processes. Some systems may require intervention in the 
machine-learning process, others require upgrades of their hardware, and 
still others require intervention in the basic software of the system. During 
this process, the al system continues its routine activity under supervision 
imposed by the court. 

Socially and functionally, probation is identical for human offenders, 
corporations, and A1 systems. The attributes of each offender require dif- 
ferent treatment, but that is true with respect to human offenders as well. 
Naturally, the manufacturers, programmers, and users of the system can 
initiate the repair process without the intervention of the court, but when 
the court orders the intervention, it means that this is the will of society. 
In the same way, a drug addict or the addict’s family may initiate a drug 
rehabilitation process without a court order, but when it is imposed by the 
court, it signifies that this is the will of society. 


6.2.4. public Service 
Public or community service is a substitute for imprisonment developed 
to ease prison overcrowding. For offenses that are not severe, the court 
may impose public service on the offender instead of other penalties, or 
in addition to them.” The offender is not incapacitated by this penalty, 
but on the contrary is forced to contribute to society in “compensation” 
for his or her involvement in the delinquency. In this way, society signi- 
fies that the delinquency is unacceptable, but because the social harm is 
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not severe, lenient measures are taken to bring about the required inner 
change in the offender. 

Public service has yet another dimension, which relates to the com- 
munity. Public service is carried out in the offender’s community to signify 
that he or she is part of that community, and that causing harm to the com- 
munity boomerangs back onto the offender.” In many cases, public service 
is added to probation in order to improve the chances of the offender’s full 
rehabilitation within the community. Public service has more than mere 
compensational value, as it proposes to make the offender understand and 
become sensitive to the needs of the community. Public service is part of 
the learning and reintegration processes that the offender experiences. 

Retribution is irrelevant for public service, because public service is 
not intended to make the offender suffer. Neither is deterrence relevant 
for public service because it is perceived as a lenient penalty with neg- 
ligible deterrent value. Both retribution and deterrence are irrelevant to 
AI sentencing. Incapacitation, which is relevant for ar sentencing, is not 
reflected in public service, unless the framework for the specific public 
service is extremely strict and the offender’s delinquent capabilities are 
incapacitated. 

The dominant purpose of public service, however, is rehabilitation 
through reintegration of the offender into society. Here is the question re- 
garding the applicability of public service: how can public service be im- 
posed on al systems? To answer this question, we again follow the three- 
stage model. First, we analyze the roots of the meaning of public service; 
second, we seek the meaning of these roots for AT systems; and finally, we 
adjust the punishment to the roots we have revealed for AI systems. 

Functionally, public service consists of supervised compensation to 
society provided by the experience of integration with society. The of- 
fender expands his or her experience with society, making integration 
easier. Broadening the offender’s social experience benefits society be- 
cause it also includes a compensational dimension. Social experience is 
not exclusive to human offenders. Both corporations and al systems have 
strong interactions with the community. Public service may empower and 
strengthen these interactions, and make them the basis for the required 
inner change. 

For example, a medical expert system, equipped with machine- 
learning capabilities, is used in a private clinic to diagnose patients. The 
system was considered negligent, and the court has imposed public ser- 
vice on it. To carry out this penalty, the system may be used by public 
medical services or public hospitals. This can serve two main goals. First, 
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the system is exposed to more cases, and through machine learning it can 
refine its functioning. Second, its work can be considered compensation 
to society for the harm caused by the offense. 

At the end of the public service term, the ar system is more experi- 
enced, and if the machine-learning process was effective, system perfor- 
mance is improved. Because public service is supervised, regardless of 
whether it is accompanied by probation, the machine-learning process 
and other inner processes are directed toward the prevention of further 
offenses during the public service. By the end of the public service period, 
the al system will have contributed time and resources to the benefit of so- 
ciety, and this may be considered compensation for the social harm caused 
by the commission of the offense. Thus, the public service of A1 systems 
resembles human public service in its substance. 

It may be argued that the compensation is in practice contributed by 
the manufacturers or users of the system, because they are the ones who 
suffer from the absence of the system’s activity. This is true not only for 
AI systems, but also for human offenders and corporations. When a human 
offender performs public service, his or her absence is felt by family and 
friends. When a corporation carries out public service, its resources are 
not available to its employees, directors, clients, and so on. This absence 
is part of carrying out the public service, regardless of the identity of the 
offender, so that al systems are not unique in this regard. 


6.2.5. f ine 

Fine is a payment made by the offender to the state treasury. It is not con- 
sidered compensation because it is not given directly to the victims of the 
offense, but to society as a whole. If we regard society as the victim of any 
offense, we may view a fine as a type of general compensation. The fine 
has evolved from the general remedy of compensation, when the criminal 
process was still between two individuals (private plaintiff versus defen- 
dant).°° When criminal law became public, the original compensation was 
converted into a fine. Today, criminal courts may impose both fines and 
compensations as part of the criminal process. 

In the eighteenth century, the fine was not considered a preferred 
penalty because it was not deemed to be as strong a deterrent as impris- 
onment.® But during the twentieth century, when prisons became over- 
crowded and the cost of maintaining prisoners in state prisons increased, 
the penal system was urged to increase the use of fines.*! To facilitate the 
efficient collection of fines, in most legal systems the court can impose 
imprisonment, public service, or confiscation of property in case of non- 
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payment.” The fine imposed is not necessarily proportional to the harm 
caused by the offense, but to the severity of the offense. 

Retribution may be relevant to fines, if the fine is proportional to the 
social harm and reflects it. Deterrence may also be relevant to fines, if the 
fine causes a loss of property sufficiently great to deter. But as noted ear- 
lier, both retribution and deterrence are irrelevant to al sentencing. Fines 
have no prominent rehabilitative value, although paying the fine may re- 
quire additional work from the offender, allowing less free time to commit 
offenses. Here is the question regarding the applicability of fines: how can 
fines be imposed on a1 systems? The main difficulty is that al systems pos- 
sess no money or other property of their own. 

Corporations possess property, so paying fines is the easiest way of 
imposing a penalty on corporations, but al systems do not possess prop- 
erty. To answer our question, we again follow the three-stage model. First, 
we analyze the roots of the meaning of fines; second, we seek the meaning 
of these roots for al systems; and finally, we adjust the punishment to the 
roots we have revealed for ar systems. The result should also include ap- 
propriate solutions for ineffective fines, that is, cases in which there are 
difficulties in collecting the fines. 

Functionally, a fine is a forced contribution of valuable property to 
society. In most cases a fine is expressed as money, but in certain legal 
systems, other valuable property may be used. In some legal systems, the 
amount of the fine is determined based on the cost of a work day, week, or 
month of the defendant, with the fine matching this cost.® Even if the fine 
is determined as an absolute sum, in most cases the absence of this sum 
in the offender’s pocket translates into the work hours needed to produce 
the missing amount. Therefore, a fine generally reflects work hours, days, 
weeks, or months, depending on the amount and on the offender’s wealth 
or skills. 

As we discussed earlier in the context of public service, the produc- 
tivity of an AI system can also be evaluated in work hours for the com- 
munity.” It is true that ar systems do not possess property, but they are 
capable of working, which is valuable and may be assigned a monetary 
value. For example, the work hour of a medical expert system may be as- 
sessed as $500. If a fine of $1,000 is imposed on an ai system, the fine can 
be translated into two work hours of the system. Therefore, the system can 
pay by using the only currency it possesses: work hours. 

The work hours are contributed to society, in the same way that public 
service is contributed. When a human offender does not have the required 
sum of money to pay the fine, other penalties are imposed. One of these 
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is public service, which may be measured by the number of work hours. 
Using work hours as a measure of payment can serve not only the pur- 
pose of paying the fine (by an accurate accounting of the number of hours 
worked), but also the optional purpose of enforcing the fine together with 
public service, imprisonment, or any other relevant penalty. 

It may be argued that this payment to society is in practice contributed 
by the manufacturers or users of the al system, because they are the ones 
who suffer from the absence of the system’s activity while it is working to 
pay the fine. This is true not only for a1 systems, but also for human offend- 
ers and corporations. When a human offender pays a fine, the absence of 
the money (or of the person, if additional work hours are needed to make 
up for the absence of money) is felt by family and friends. When a corpora- 
tion pays a fine, its resources are not available to its employees, directors, 
clients, and so on. This absence is part of paying the fine, regardless of the 
identity of the offender, so that AT systems are not unique in this regard. 


closing The opening exaMple: sentencing The Killin g r oboT 


We opened this chapter with an example that described an ai robot con- 
victed of manslaughter. We can now return to this example and analyze 
it based on the insights gained from this chapter. After the court has con- 
victed the a1 robot for manslaughter, the criminal process continues as if 
the offender were human, and the next stage is that of sentencing. The 
prosecution and the defense may argue for a harsh or lenient penalty, and 
eventually the court decides the appropriate penalty based on the relevant 
circumstances of the case. 

In our example, after assessing the circumstances, the court reaches the 
conclusion that if the robot were human, the relevant penalty would be 
three-year prison sentence. If the robot had been a corporation, the court 
would have imposed three years of imprisonment, which would have been 
carried out as demonstrated earlier.” The same should be the case with 
the AT robot. Therefore, if the judicial discretion of the court results in the 
imposition of three years’ imprisonment, this is the penalty that must be 
imposed, regardless of whether the defendant is a human, a corporate, or 
an al offender. 

After its imposition, the penalty must be carried out. As previously 
noted, the imposition of imprisonment on AI systems takes the form of 
depriving them of their liberty to act by restricting their activity for the 
required term and under strict supervision. During this time, the al system 
can be repaired to prevent the commission of further offenses. Repair of 
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the ar system may be more efficient if the system is incapacitated and 
when it happens under court order. This situation serves both the pur- 
poses of rehabilitation and incapacitation, which are the relevant sentenc- 
ing purposes in the case of al systems. When the al system is in custody, 
restricted, and supervised, its capabilities to offend are incapacitated. 

Repairing the a1 system through inner changes, initiated by external 
factors (for example, programmers working under court order) and experi- 
enced by the system as a restriction, is substantive rehabilitation because 
the chances of the system being involved in further delinquency are re- 
duced. The social value of imposing imprisonment on an a1 system is real. 
The dangerous system is being removed from society for repairs, during 
which time it is not capable of causing further harm to society. When the 
process is complete, at the end of the term determined by the court, the 
system can be returned to full activity, the same way that a prisoner who 
finishes serving a penalty is returned to society. 


conclusion $ 


Criminal liability for artificial intelligence entities may sound radical. For 
centuries, criminal liability was considered to be part of an exclusively 
human universe. The first crack in the concept occurred in the seventeenth 
century, when corporations were admitted into this exclusive club. Cor- 
porate offenders joined human offenders, as both criminal liability and 
punishments were equally imposed on both. The twentieth century intro- 
duced ai technology to humankind. This technology developed rapidly in 
an attempt to imitate human capabilities. Today, a1 systems imitate some 
human capabilities perfectly; indeed, they outperform humans in many 
areas. Other human capabilities, however, still cannot be imitated. 

Criminal liability does not require that offenders possess all human 
capabilities, only some. If an al entity possesses these capabilities, then 
logically and rationally, criminal liability can be imposed whenever an 
offense is committed. The idea of criminal liability of a1 entities should 
not be confused with the idea of moral accountability, which is a hugely 
complex issue not only for machines, but for humans. Morality, in general, 
has no single definition that is acceptable to all societies and individuals. 

Deontological and teleological morality are the most acceptable types, 
and in many situations they lead in opposite directions, both “moral.” The 
Nazis considered themselves to be deontologically moral, although most 
other people, societies, and individuals thought otherwise. Because moral- 
ity is so difficult to assess, moral accountability is not the most appropriate 
and efficient way of evaluating responsibility in criminal cases. Therefore, 
society has chosen criminal liability as the most appropriate means for 
dealing with delinquency. Modern criminal liability is independent from 
morality of any kind, as well as from the concept of evil. It is imposed in 
an organized, almost mathematical way. 

Criminal liability has definite requirements, not more and not less. If 
an AI entity meets all these requirements, then there is no reason not to 
impose criminal liability on it. We have seen that AT entities are capable 
of meeting these requirements in the same way that human and corporate 
offenders do (but not animals). We have also seen that all types of crimi- 
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nal liability are relevant to A1 technology, and all substantive arguments of 
criminal liability can be made regarding ai entities. Finally, we have also 
seen that penalties can be imposed on ai entities in a way that substan- 
tively resembles the way in which corporations are being punished. 

Society derives the usual benefits of criminal law by imposing crimi- 
nal liability on AT entities. The conclusion is simple. Either we impose 
criminal liability on AT entities, or we must change the basic definition of 
criminal liability as it developed over thousands of years, and abandon the 
traditional understandings of criminal liability. 
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